Specifying a protein substitution matrix
You can specify your own substitution matrix by using the -matrix option. This reads a protein substitution matrix in NCBI or WU-BLAST format. The alphabet is assumed to be amino acid, and sum-of-pairs scoring is used. The ­-gapopen, -gapextend and -center parameters should be specified; normally you will specify a zero value for the center. Note that gap penalties MUST be negative. The environment variable MUSCLE_MXPATH can be used to specify a path where the matrices are stored. For example,

muscle -in seqs.fa -out seqs.afa -matrix /data/matrix/blosum62 -gapopen -12.0 -gapextend -1.0 -center 0.0

Example matrices can be downloaded from the NCBI FTP site. At the time of writing they are found here:

                ftp://ftp.ncbi.nih.gov/blast/matrices/

Specifying a nucleotide substitution matrix
MUSCLE isn't really designed to support a nucleotide matrix, but you can hack it by pretending that AGCT are amino acids and making a 20x20 matrix out of the original 4x4 matrix. You MUST specify the -seqtype protein option to fool MUSCLE into believing that it is aligning amino acid sequences.The ACGT rows and columns should have your desired nucleotide values, the other entries should be zero. Values can be integers or floating-point. In the following example, integers are used with a match score of +1 and a mismatch score of -2 (these are the BLASTN defaults).

Download an example matrix here.
 
Use the following options:

    -matrix <filename> -seqtype protein