| Articles Home |
MNSure.org No Testing or QA, MN.IT Services FAIL
MNSure Security Problems
MNSure Bait and Switch Month Long Backlog
MNSure Citizen Reject Ripoff Private Insurance
MNSure Redesigned to use Prescreening
MNSure PR Propaganda
MNSure Minor Fixes to Up Enrollment
MNSure Enrollment Lags Compared to Successful State Exchanges
Minneapolis Star Tribune Health beat: Data guy questions MNsure system
Dec 12, 2013
It is mid December and the Federal Obamacare web site, HealthCare.gov, had its upgrade and can handle over half a million users per day. Performance has improved, usually easy fixes, even if there are still lots of data bugs.
However, as an enrollee in the MCHA (MN Comprehensive Health Assoc.) I have to use MNsure.org, the state version of an Obamacare health insurance exchange. The State of MN told me I am being dropped from the state MCHA program where YOU pay %25 over market after you are dropped from commercial health insurance scams for pre-existing conditions. My condition? As near as I can figure it is that I am over 50 years old. When I enrolled in MCHA I filled out a form and got billed a lot, seemed pretty easy after surviving the 90 day mandatory gap in coverage where they hoped you dropped dead.
MNsure.org has spent about $50 million per year and has another $40 million to spend for a total so far of over $150 million. The web site performance is like molasses in January and there are no real numbers of people publicly released who actually got insurance from the MNsure.org web site "enrollment".
As a user of MNsure.org trying to find a replacement for my MCHA insurance I am not through the process yet, slow web pages don't help, I have spent 7-8 hours on the site so far, it is frustratingly slow and difficult. Back in mid October I could not even get an account after hours of trying multiple times over several weeks. Down time over weekends and after "business hours" didn't help.
The user interface design is so FAIL it is embarrassing to be from Minnesota. In October the plans and costs were hidden so deep in the site that I had to contact my legislator, Phyllis Kahn, to get the link to see the plans without an account after I had emailed the contact address at MNsure.org but got no reply to my questions. It still does not show where to shop for insurance plans from the home page, and when you find the plans they all show up at once, 70+ ways to pay and pay in insurance gibberish that I do not understand.
Basic MNsure questions no one will answer:
I found a Pew Charitable Trusts December 10, 2013 article about the "working" state insurance exchange websites vs. the 10 websites that are not "working". The theme of the article was "Simple and Well Tested". I decided to check the article's hypothesis and compare the "working" sites (WA, CT, KY, RI) with MNsure using performance and code tests to see if the "working" websites used performance enhancements and had fewer errors than MNsure.org.
To see what was going on with the web site performance I ran performance tests on the MNsure homepage with free online tests at http://www.webpagetest.org and http://www.websiteoptimization.com/services/analyze/ .
Surprise, surprise I found a bunch of simple stupid problems. The problems are so stupid it is evidence that web site performance has not been a factor in the design of the project. When basics are ignored a pattern emerges of incompetence, no quality control, no acceptance criteria from the State of Minnesota MNsure managers. The result is garbage code from scamming contractors.
--https://www.MNsure.orgOf course, most websites can fix these problems with a few hours of competent work with a system administrator that knows his business. Configuration file changes to cache and compress files and a few thousand a year for a CDN contract is not rocket science. Coding standards of where to put scripts are easy to establish but are tedious code monkey work to fix later. Stripping out larded up 3rd party commercial sites that can hang your page if they have outages out of your control only makes sense if you want to offer life or death products, like say, HEALTH CARE, it takes no expert skill to do this step, only a demonstration of commitment to the welfare of citizens.
It would have helped if web site performance had been thought about 2 or 3 years ago as any competent web project manager would have done.
Did anyone implement performance enhancements in any of the "working" health care exchanges? Why yes they did, the successful ones did anyway, and even the Feds.
I checked some pages on the well known standards site w3.org using online tests for CSS and HTML errors comparing the successful state websites to MNsure.org. Are a couple errors fatal? No, but a whole bunch of errors can indicate sloppy or incompetent work.
Again, it looks like there is no quality control set up by Maximus or any acceptance criteria by the State of MN.IT Services to receive and implement error filled crapware before foisting it on the public. Do you see the pattern yet? The pattern of incompetence by both the contractor and the MNsure management, paying for error filled broken code without acceptance criteria for IT contracts. A bunch of amateurs put in charge of a rapacious contractor is a set up for graft and problems.
URLs tested:The pattern should be clear by now, the State of Minnesota has problems managing IT contracts and the contractor stinks.
KSTP Story on Delayed Reply to Legislature of a Letter sent from MNsure Dec 15
Quote from the MNsure Letter to the Legislative Oversight Committee:
" Has there been user-testing (demo or beta testing) for public consumers
of the Insurance Exchange system prior to October 1?
Due to the short window to develop MNsure, consumer testing was not done. However,
we plan to perform consumer testing after the first open enrollment period is done."
MNsure payments of $7,139,414 went for testing to Maximus including "user testing". It seems that "user testing", an item in the contract, did not get done but was paid for. Is that what happened?
January 7, 2014 I noticed on plans.mnsure.org I get mixed http and https content in a Firefox warning for a "man in the middle" attack. Note that plans.mnsure.org is different than www.mnsure.org and they should also check for Mark Lanterman's different "man in the middle" attack on this subdomain.
January 12, 2014 I noticed the same Firefox "man in the middle" problem on https://payment.mnsure.org/financials, the sign in page, which is not a good page for security problems.
Feb 1, 2014 https://auth.mnsure.org/appeals/appeals-information-page.jsp same Firefox "man in the middle" problem.
Netcraft.com shows a
security risk,
in its report on MNsure.org. I wonder what it is?
Legislative Auditor, please ask some questions of Netcraft.com, we have already had data leaks
noted in the media as covered by your official report on that subject.
UPDATE: Feb 21, 2014 Netcraft shows no risks at MNsure.org
These "working" state websites and even the Feds had no risk at Netcraft. Healthcare.gov, RI and KY
Another security risk is the contractor Maximus. Maximus, the lead contractor, has had health care billing scandals in the recent past. Maximus agreed to pay $30.5 million to settle Medicaid fraud criminal charges in Washington DC. I saw nothing in the Risk Management documents on the MNsure.org site that showed they paid attention to Maximus' legal problems or the many failed government projects of Maximus.
So where did the $150 million go? I looked up the RFPs and payment contracts at MNsure. The tale is so sad. Some real expensive software was sold to MNsure and a whole pile of licensing fees show it. Why licensing was paid for non-working software , such as IBM Curam, is a mystery. Again it shows the software was paid for without validation and acceptance testing.
Expensive testing seems to have been paid out starting in Feb 1, 2013 in $2 million+ chunks every couple months until September for a total of $7,139,414. That is what I do not understand, on paper it looks so clean with everything tested and paid for, but the MNsure website from Oct 1 to now shows that testing did not happen and certainly not the "fixing".
MNsure.org seems to have paid without acceptance criteria or internally testing the code from Maximus and other contractors. It sure looks like Maximus didn't test anything. The contracts claim the work was done but a website that is open less than Target (which is open weekends) and filled with errors shows it was not. I think they need an audit, maybe there can be some clawback.
On Oct 9, 2013 they changed some servers or software. It looks like they bought some expensive software packages from F5.com. The company touts its BIGIP products for "performance" but since the performance sucks and they made flailing web server changes in mid October they look like they bought some expensive software that did not work or they did not know how to work it.
Another guess I would make is that they bought top of the line expensive production web servers in 2011 that in 2013 look like a herd of turtles dying from old age. This is usually done so the contractor gets hardware profits early in the process.
In mid December I can see that MNsure.org is still not doing it right, problems in getting the information to the insurers (thousands of people), mistakes in thousands of accounts that were assessed incorrectly to not get insurance premium assistance. I bet some real money has gone to scam contractors that created an elaborate unproven MNsure.org architecture that does not play well together but had to have every bleeding edge tech buzzword included for a big front end payout. That way the contractor gets to skim software profits right away and if they are fired they still get paid. The result is that the MNsure.org site still goes out of service on week nights and weekends.
The government of Minnesota is not the only one that just failed on a big IT implementation. Avon the cosmetics company, just announced a 650 person layoff and a writeoff of $125 million on a failed 4 year Ipad SAP sales application. Private industry also has the dead bodies of large IT contracts littering the computer landscape.
The point is that big complex IT systems projects are hard to do, Minnesota did not bring in its A game. April Todd-Malmlov an exec from United Health who did mostly public relations and was an "economist" at the state knew nothing about software projects so why did they put her in charge? The rest of the top management is about the same, lawyers, DHS managers, hospital and insurance execs, I notice only 2 of fifteen MNsure administrators have any IT experience in their biographies and that they seem to have not been effective in their MNSure positions, in fact they seem exceptionally weak. A third of the money was spent on a software project, I would have expected a stronger team from MN.IT Services at MNsure, maybe a third or more of the people in management alone.
The IT people from the MN.IT. Services were weak, whether they were ignored by the management, bullied by contractors, just incompetent or feared losing their jobs it shows that IT.MN Services can't handle problems or have any positive effect when a project starts to go pear shaped. MN.IT Services just rode the MNsure.org project straight into the ground and is shocked, shocked to learn it is all a disaster when they should be the front line to keep the contractors on task and raise early warnings when problems arise.
I sure wish I could just get Medicare for All or MNCare for All. Medicare systems have been set up for over 50 years and work. Then we would not need to pay for $150 million websites for each state or the hundreds of millions for the HealthCare.gov site or the 15-20% profit for the insurance companies. Maybe we could pay for health care for all citizens on the FICA payroll taxes instead of using rip-off crook contractors and insurance scams.
Most of the people without insurance can't afford the up front payments until you get your taxes filed in April or August to get your "tax credit". 4 or 8 months of premiums for a family will costs many thousands. If uninsured people had that much money in the first place they probably would have insurance. And there is no such thing as a "late" payment in the insurance business, you immediately lose coverage, then you can reapply to get it back. You could spend $5,000 for several months insurance, get sick, miss work, send a late payment and you are uninsured with huge medical bills with the privilege of reapplying to pay thousands more. And my experience is that your late payment is lost money, you have to know how to get it back from the insurance company with lawyers, they don't just give it back to you, how many people will figure all that out? This ACA insurance industry bailout is a massive extraction of wealth from poor people.
My folks have Medicare and a Medi-gap policy. No ACA-gap insurance is offered. When all those 60%-90% policies generate a bunch of medical bills that wipeout to bankruptcy every policy holder that runs into a serious health problem... You do the math. It is just more wealth extraction.
But as the State of Minnesota we didn't give our citizens health care, instead we gave a billion to Ziggy Wilf for a stadium and $150 million for a crappy web site and billions more to insurance companies. The billions in profit given to insurance companies is a tax on citizens just like the tax money given to Ziggy Wilf writ even larger, remember that United Health Care executives have had payouts of over a billion dollars per year. It's like buying new Ziggy stadiums every year. Forever.
-- https://www.mnsure.org HTML validation on validator.w3.org
62 HTML errors, 58 warnings found. December 13, 2013, 404 error found.
71 HTML errors, 69 warnings found. December 31, 2013, 404 error found.
Crash Thursday Jan 2, 2014, outage part of Fri,Sat,Sun,part of Mon.
74 HTML errors, 72 warnings found. January 6, 2014, 404 error found.
The testing above shows that the MNsure is trending in an interesting way: code gets more errors and the system seems to be losing stability. MNsure did not announce any "fixes" to any code when it came back Jan 6, 2014 after being down parts of 5 days. The size of the largest Javascript file, https://www.mnsure.org/JS/mn_jquery-ui-custom.js seems changed in size from 7 days before.
Over the last few years there has been a consolidation of IT services in MN. A plan was drawn up in 4 phases to centralize control of IT people in Departments and create Department CIO administration. The final "Phase 4" is now in effect.
According to the Consolidation Plan in IT.MN Services there are many wonderful things realized, including undescribed amorphous "best practices", advertising "success", and of course myriads of Chief Information Officers who have Department responsiblity for all things IT. Unfortunately in all the 4 plan phases I saw nothing about contract management, acceptance criteria for software and hardware, authorizing payment of contracts to vendors and contractors after quality assurance and acceptance criteria are met. Nothing about vetting the processes of vendors and contractors to assure that installation of hardware, development systems, packaging of software, testing and maintenance processes are in place to assure success of projects.
Tactical Plan FY 2013-14 says fluffy phrases about some sort of procurement management, but I see no actual documents reflected in the project management policies. Mostly the Tactical Plan is "we want cloud", but not for any specific use. If MN.IT Services wants a "cloud" they should have a specific project or 3 to put on the cloud. Most financial and procurement management detailed policies say approximately: "if it is expensive it has to be signed off by a bigger boss than you".
In fact in all MN.IT Services project management documents I saw a hollow paper trail of nothing. Risk Management is talked about but no details about what a risk is or what acceptance criteria should be checked off for hardware or software contractors/vendors to minimize risk. There are some pre-approved licenses and materials but that is it, for example, you can get a desktop and some Microsoft applications software like Excel for an employee. No real software engineering practices concerning development, configuration management, security, QA or maintenance are covered in the project management documentation. It looks to me that a very flexible project management scheme is in place with no controls of vendors/contractors or anything, really, just fill out some simple forms and start paying invoices.
The MNsure project is managed very successfully on paper. The project plan, risk plan, work plan and payment schedule were set up and at least the payment schedule was followed rigorously. But no one looked to see that the vendor did the work and no work was done by MN.IT Services to verify the work. Central MN.IT Services lawyers should be reviewing every IT contract to keep the state from being ripped off.
The role of MN.IT Services is central in the pervasive problems of software projects in the State of Minnesota. Now that the consolidation of state IT employees has been accomplished we see no benefit. When a project fails the Department CIO continues in the job or even promoted. Maybe it is time for regime change at the top of MN.IT Services and proper development and vendor/contractor standards are put in place for all MN.IT Services projects.
It seems MNsure knew the risks of using Maximus and cut them out of the picture in Feb 2013, (J.Nord -MinnPost), they just did not put it in the "Project Plan Risk Management Plan" or tell anyone about it. But MNsure still seemed to pay Maximus.
MN.IT Services stepped in, but how? They have some system administrators to fire up the servers, but how many software engineers do they have working on MNsure? Are they using accepted software development practices followed by every program available on the internet, code repositories, bug tracking, quality assurance, packaging the software for easy installation? I see no evidence of that. I see the Governor's famous 21 point bug list and a short list of hints(bugs) about how to write your name without a hyphen when creating an account.
The 'Man in the Middle' security issue raised by Mark Lanterman may still be present on https://plans.mnsure.org/mnsa/planadvisor/plan_advisor.htm?flow=anonymous . I get mixed http and https content in a Firefox warning that a "man in the middle" attack is possible. Note that plans.mnsure.org is different than www.mnsure.org, it is a subdomain. On https://www.mnsure.org/ I get no such warning. If it is not the same problem as seen by Mark Lanterman on www.mnsure.org it is still a problem and should be fixed.
Also the same security problem is seen on https://payment.mnsure.org/financials, a sign in page, mixed http and https, Firefox warning of "man in the middle attack" possible. This subdomain should also be checked for the Mark Lanterman security issue.
At the MNsure Legislative oversight committee we were told of the famous Governor Dayton IBM bug list that bug 10 (of 21) is not fixed. Leitz claims 6 are fixed, 8 are in being coded and in test, 7 have no solution yet but except for bug 10 no one but MNsure knows which is fixed. They should release a bug list like open source projects.
January 16, 2014 https://people.mnsure.org/ discovered as a subdomain, a Curam icon and Curam Title comes up, but the page is blank except for a top banner with no links. https://auth.mnsure.org/ also found as a subdomain. So far www.mnsure.org, plans.mnsure.org, payment.mnsure.org, people.mnsure.org, auth.mnsure.org, dpr.mnsure.org, id.mnsure.org.
2 opposite status in my application so far, latest status says MNsure will contact me and tell me something, I do not understand what it says. It has been 4 days, no contact, I can't figure out what my status is and MNsure has thousands, maybe tens of thousands of people in limbo. Guess what, I am not going to call and be put on hold for hours, not gonna happen.
Maybe my status is that they are tired of me pounding on the website, I left an error log trail of dozens of 404's, 403's and 500 errors trying to finish my application while not even trying to deviate from the straight and narrow path because I was too afraid to crash out. It is so obvious that no one is looking at the error logs to fix the missing pages, links to garbage URLs, bad scripts, etc.
So now after starting in October I am in the famous "Black Hole" with the rest of the suckers who have wasted days trying to apply. It's a wonder anyone finished the application process at all.
MNsure finally got $162K logo on its website icon, was broken on the home page since Oct1. Good to see progress! But that is the only error fix I have noticed.
MNsure JavaScript errors discussed about live code on website: http://www.reddit.com/r/webdev/comments/1svjyk/in_my_world_you_dont_get_to_keep_your_job_mnsure/ refers to: http://imgur.com/5C3FSyR
More javascript errors on home page:
Serve resources from a consistent URL
The following resources have identical contents, but are served from different URLs. Serve these resources from a consistent URL to save 1 request(s) and 203.5KiB.
https://s7.addthis.com/static/r07/core120.js
https://s7.addthis.com/static/r07/core120.js?_=1391445254630
The following resources have identical contents, but are served from different URLs. Serve these resources from a consistent URL to save 1 request(s) and 6.6KiB.
https://s7.addthis.com/js/250/addthis_widget.js
https://s7.addthis.com/js/250/addthis_widget.js?_=1391445254629
I found a phone number referenced in a news article that was not the MNSure phone bank, it was at DHS (Dept. of Human Services). Everyone else was getting 60-90 minute waits at MNsure and when connected many got no help. My connection time to a live person was about 10 seconds. The person on the other end of the line was a grizzled veteran, looked up my case on MNsure, sent it for manual handling and several weeks later I got a card. It is my understanding that without the phone call and the manual handling of the case my finished applications would still be stuck in the system. 33 days total from finishing my application to getting a card.
It took months though to finish the application because I could not even get an account the first months and then I did not finish the forms in one session but spaced it out over many weeks.
I read hundreds of pages of documents, RFP's, MNsure vendor contracts, news stories, 2 sets of internal IT review reports from 2 different vendors, MN.IT Services contract and project management documents, Minnesota State Statutes and Rules that cover MNSure and MN.IT, Legislative Auditor reports including many on the failed Governor Pawlenty DHS "HealthMatch" project which is closely related to MNsure.org. I tracked the Oregon website scandal, including the probable faking of Federal benchmarks related to continued Federal funding, which may have also happened at MNsure. I saw tens of hours of MNsure Board meetings, Legislative Committee meetings. I called navigators, insurance brokers, friends, politicians. I compared code validation and website performance settings to other states health exchange websites, examined javascript for errors and utility, investigated 3rd party javascript included on MNsure.org. I investigated MNsure employees records on the internet, tracked scandals of MNsure related politicians.
So yeah, I know something about MNsure and I don't like what I see. It is not a simple system for single payer health care, it is a travesty.