Nowadays, the research in deoxyribonucleic acid (DNA) cryptography seeks to implement data transmission techniques to ensure secure data transmission across the world. As data transmission techniques are not secured due to the presence of hackers and attackers, a DNA-based cryptosystem can be suitable to secure data transmission, where confidential information (plaintext) is encoded in an unreadable form (ciphertext) prior to its transmission. This paper proposes a novel cryptosystem based on DNA cryptography and finite state machines. Here, finite state machines perform substitution operations on the DNA sequence and make the system more secure. Moreover, a DNA character conversion table is proposed in this paper to increase the randomness of the ciphertext. The efficiency of the proposed scheme is tested in terms of the randomness of the ciphertext. The randomness of the ciphertext determines the security of a cryptosystem, and here, randomness tests mentioned in the National Institute of Standards and Technology (NIST) test suite assess the randomness of the ciphertext. The experimental results show that the proposed scheme yields an average P-value of 0.95, which outperforms the existing systems. The proposed scheme guarantees a highly secured cryptosystem as an average avalanche effect of 75.65% is achieved. As a result, the proposed scheme is more secure than the existing DNA-based cryptosystems.