It looks like you have not lived in Colorado for at least 6 months and the child does not live in Colorado right now, therefore there is a problem with using the Colorado courts.
I'm assuming that when you say "agreement" that the document is actually a court ordered parenting plan and not a private agreement. You need to go back to the attorney who represented you in the divorce and have the Kansas order amended to state that you can take the child to your home in Colorado. Clearly, your ex seems to be a jerk, but you cannot get into an altercation over the child at his house or you'll get arrested. Moreover, if the "agreement" is a court order, then you violate the order by varying from its exact terms.
If the "agreement" was a private agreement, then I would recommend that you go to court where the child is residing (seems like Kansas) and file a petition to determine parenting rights and child support (and possibly for divorce). It is important to protect your rights and clearly if you have a private agreement it isn't worth much if your ex isn't going to be reasonable. You may end up paying the ex depending on your respective incomes and the time that the child spends with each of you.
In any event, an hour of a lawyer's time should give you a good idea of your rights and the recommendation as to how best to proceed.
Some attorneys sell unbundled legal services where you pay for just what you need and can afford. For example, some attorneys will give you an hour or two at a set price to review your lease and give you advice based on the law, prepare letters for you to sign, or sell the paperwork for the court filings; then you can proceed on your own, but knowing that your paperwork is correct.
Good luck. jim
NOTICE— The Legal Information contained in this answer is provided solely for informational purposes to assist the layman with understanding legal issues, and is not legal advice nor intended to create an attorney-client relationship between you and me or my law firm. Such Legal Information is intended for general informational purposes only and should be used only as a starting point for addressing your legal issues. It is not a substitute for an in-person or telephone consultation with an attorney licensed to practice in your jurisdiction about your specific legal issue, and you may not rely upon such Legal Information. You understand that questions and answers or other postings to the Site are not confidential and are not subject to attorney-client privilege.