The site went offline on Thursday.
Liberty Reserve had described itself as being the internet's "oldest, safest and most popular payment processor... serving millions all around a world".
It had allowed users to open accounts and transfer money, only requiring them to provide a name, date of birth and an email address.
Cash could be put into the service using a credit card, bank wire, postal money order or other money transfer service. It was then "converted" into one of the firm's own currencies - mirroring either the Euro or US dollar - at which point it could be transferred to another account holder who could then extract the funds.
The service promised that payment transfers were "instantaneous" and it charged a maximum of $2.99 (£1.98) for each transaction. It also offered a private messaging facility which it said was "much more private and secure than email or instant messenger services".
Security expert Bryan Krebs said Liberty Reserve's features had made it a popular among cybercriminals who wanted to move funds and make payments anonymously.
However, others said they had used the service for legitimate means, viewing it as a cheaper alternative to PayPal. They fear they will now lose money still sitting in its accounts.
Refused licence
Costa Rica state prosecutor Jose Pablo Gonzalez announced news of the raids on Saturday. He said that in addition to Mr Budovsky's arrest, 10 other suspects were being investigated in connection with international money laundering.
Liberty Reserve Liberty Reserve
Mr Gonzalez said that Costa Rica's financial regulator, Sugef, had refused to issue a licence to Liberty Reserve in 2011 after raising concerns about how it was being funded.
Although the company Liberty Reserve subsequently closed and dismissed its employees, the prosecutor said Mr Budovsky had continued operating his money exchange service by running it through five other Costa Rican businesses.
The same year, he said, New York based officials asked Costa Rica to investigate.
Mr Gonzalez said offices and houses linked to Mr Budovsky had been raided last week, and that documents and three vehicles - including a Rolls Royce - were seized.
The US Department of Justice's website says that in 2006 Mr Budovsky was found guilty of operating a separate illegal money transmittal business - GoldAge - from a Brooklyn apartment when he used to live in New York.
However, when asked about the latest arrest, a spokeswoman said she could "not provide any guidance" at this time.
The BBC has also asked the Spanish police for more details.
