Prostitution is a victimless crime - a voluntary and consensual transaction between informed adults. Accordingly making it a crime is immoral. While a client of a prostitute may be breaking a promise to a partner and perhaps breaking a marital promise, this is not a crime. Even if this were a crime, this would not make prostitution a crime for the prostitute, or an unpartnered client. Punishing someone for the actions of others is immoral.
See