You can develop four different models to predict the readability score:
just based on the word
based on the word and text colour
based on the word and font type
based on the word, text colour and font type
This kind of modelling works best if you reserve some words that you do not use initially to create your model. Once you have created a model, then try it out on these words to see how well your predictions match the actual readability scores.