The purpose of this study was to investigate how the subjective impression of English speech would change when pause duration at punctuation marks was varied. Two listening experiments were performed in which written English speech segments were rated on a variety of evaluation items by both native-English speakers and non-native speakers (native-Chinese speakers and native-Japanese speakers). The ratings were then subjected to factor analysis. In the first experiment, the pauses in three segments were made into the same durations, from 0.075 to 4.8 s. Participants rated the segments on 23 evaluation items on a rating scale from 1 to 10. A varimax rotation after PCA (principal component analysis) led to two factors that were related to speech style. These two factors could be interpreted as representing speech naturalness and speech rate. Speech segments with a pause duration of 0.6 s received the highest naturalness evaluation, while perceived speech rate decreased as the physical pause duration increased, without any changes in utterance segments. In the second experiment, a full-factorial design of pause durations (0.15, 0.3, 0.6, 1.2, and 2.4 s) within and between sentences, i.e., for commas and for periods, was implemented in two speech segments. The original speech segments and speech segments without any pauses were also included as control conditions. From ratings on 12 evaluation items, similar to Experiment 1, two factors representing speech naturalness and speech rate were obtained. The results showed again that the perceived speech rate decreased with an increase only in pause duration. As for speech naturalness, the highest evaluations occurred when pause durations were 0.6 s within sentences, and either 0.6 or 1.2 s between sentences. This recommends fixing all pause durations to 0.6 s as a practical way to train non-native speakers to make their spoken English appear more natural.