Login | Register

Coinbase online payment system

Coinbase
https://coinbase.com/

Updated on October 11, 2012
Views: 6590 | Clicks: 272


Website Screenshot



General Information

 

 

Glyphicons_055_stopwatch
Instant Payments

Payments arrive at the speed of an email (just a few seconds) and are confirmed within the hour. No more waiting three business days for checks.

Glyphicons_037_credit
Low Transaction Fees

Coinbase charges just 0.5% when you buy or sell bitcoin via bank account transfer. After that all bitcoin-to-bitcoin transactions are free.

Glyphicons_163_iphone
Pay By Phone

Our website works great on modern smartphones (iPhone, Android, etc). Just visit coinbase.com from your mobile browser.

 

Glyphicons_263_bank
Simple Transfers

Use your bank account to purchase bitcoins. Transactions are processed within two to three business days. (coming soon)

Glyphicons_280_settings
Merchant Tools

Easily create "buy now" or donate buttons. We also offer full shopping cart integration. (coming soon)

Glyphicons_040_stats
Widespread Adoption

About $2 million a day (USD) is already being transacted in bitcoin. It's quickly becoming an international currency.

Currencies

Bitcoin

Countries of use

USA

Users

private and business

Fees

0.5% when you buy or sell bitcoin via bank account transfer

Recent news

Posted on February 17, 2019
Electrohunt Part 1: Hunting for the phishing campaigns on the Electrum network

Users of the Electrum Wallet, a popular desktop Bitcoin client, have been under a persistent attack since December of 2018. According to Cointelegraph, it resulted in a million dollar loss worth of crypto in just its first few hours.

The sheer size and impact of the attack got me interested in learning more about it. Most of the discussion in the news has been focused on the vulnerability in the Electrum Wallet software which allowed arbitrary phishing messages to be displayed. The Electrum Wallet software is unique as it relies on a separate distributed network of specialized servers acting as gateways to the Bitcoin network. As I have discovered, it is that supporting network of Electrum servers which is under a constant attack by bad actors which can lead to even greater impact on the users than the current phishing campaign.

In this blog post, I will share my research into how the attackers have exploited the trust model in the Electrum network and flooded it with malicious servers. I will also share the story of my hunt to discover two ongoing phishing campaigns, reveal details about the attackers’ infrastructure, and their unique methodology to backdoor Electrum Wallets.

The Phishing Attack

The client phishing attack has been covered by a number of sources listed in the reference section below so I will only cover technical details relevant to the exploration of the networking portion. Electrum Wallets prior to version 3.3.3 had a unique feature where error messages sent by the server would be displayed in a pop up dialog. The original intent was likely to inform users of the reason for their transactions not going through. However, exposing users to arbitrary messages from an untrusted network of servers operated by unknown parties is ripe for exploitation.

Attackers have learned to modify the open source ElectrumX server software to always send arbitrary error messages. The screenshot below illustrates an error popup message that I received on a vulnerable Electrum Wallet:

Following the website in the error message leads to a phishing site with several versions of the Electrum Wallet available for download. After the user downloads a fake update, the backdoored versions of Electrum Wallet will steal user’s funds.

So what are those servers? How do attackers manage to force a wallet to connect to them and display malicious messages? To answer these questions, I had to step back and explore how the Electrum network is built and figure out its weaknesses.

The Electrum Network

As a light bitcoin node, the Electrum Wallet does not maintain the entire state of the Bitcoin blockchain. Instead, the Electrum Wallet relies on a distributed network of gateway servers which can query the complete blockchain state and transmit transactions on its behalf.

One implementation of such gateway server is called ElectrumX. The ElectrumX server builds on top of a full Bitcoin node and introduces a number of convenience functions to help Electrum Wallets interact with the blockchain. The diagram below illustrates the relationship between an Electrum Wallet client, ElectrumX servers, and the rest of the Bitcoin network:

The ElectrumX servers form their own distributed network to increase resiliency to individual nodes going offline or falling behind the blockchain. The network relies on anonymous volunteers who get no financial incentive (e.g. there are no transaction fees). Electrum servers actively discover, advertise, and maintain connections with other servers through a peering process. As a new ElectrumX server comes online, it can announce itself to the rest of the network and start accepting connections from Electrum Wallets. Here is a snippet from the Peer Discovery specification describing the process:

Server.add_peer
:func:`server.add_peer` is intended for a new server to get itself in the connected set.
A server receiving a :func:`server.add_peer` call should not replace existing information about the host(s) given, but instead schedule a separate connection to verify the information for itself.
To prevent abuse a server may do nothing with second and subsequent calls to this method from a single connection.
The result should be True if accepted and False otherwise.

The way that the add_peer verification process is implemented in practice is that the server simply verifies that the provided peer hostname resolves to the IP address of the node attempting to join the network.

When the Electrum Wallet first connects to the network it has a hard-coded list of trusted servers. However, after it establishes the initial connection and as a user connects to different servers, it actively discovers and saves additional Electrum nodes by asking them for their peer sets (servers they know about). For example, the image below shows the Electrum Wallet network configuration dialog and all of the discovered peers which may be used to connect to the network at a later time:

As Electrum Wallet discovers new servers and their peers, they are added to the local storage and depending on a user configuration may be used the next time the client connects to the network. By design, the Electrum Wallet attempts to maintain connections to a randomized selection of discovered servers on the network.

In the case of a phishing attack described above, the Electrum Wallet would have to discover and connect to a malicious ElectrumX server which in turn would trigger a popup dialog. The more malicious ElectrumX servers in the network, the higher the chance that a wallet will be attacked. In order to understand exactly how an attacker can do that, I needed to better understand how clients and servers communicate over the Electrum network.

Communication Protocol

The communication between ElectrumX servers and wallets uses a protocol based on JSON RPC. The protocol relies on a collection of verbs which can be used to obtain server properties and interact with the blockchain. Here is a sample exchange between a client and a server to obtain a server version:

Client ->
{“id”: 3, “method”: “server.version”, “params”: []}
                                                        <- Server
{“jsonrpc”: “2.0”, “result”: [“ElectrumX 1.9.5”, “1.2”], “id”: 3}

In the above exchange, I used the verb server.version to which the server replied with a string “ElectrumX 1.9.5”. Another common interaction between both servers and wallet clients is to obtain a list of peers. In order to simulate a fake wallet software, I used the several examples and libraries in the excellent connectrum python module:

$ python cli.py --server electrum.hsmiths.com --port 50001 --protocol t server.peers.subscribe 
Connected to: electrum.hsmiths.com
Method: server.peers.subscribe
[
[
"5.230.24.38",
"daedalus.bauerj.eu",
[
"v1.4.1",
"s50002",
"t50001"
]
],
[
"81.7.13.84",
"81-7-13-84.blue.kundencontroller.de",
[
"v1.4.1",
"s50002"
]
],
[
"144.76.99.209",
"ecdsa.net",
[
"v1.4.1",
"s110"
]
],

The response contains a nested data structure with a complete list of every server that this node is aware of. Let’s break down individual lines to see what they mean:

[
"5.230.24.38”, <- server IP address
"daedalus.bauerj.eu”, <- server hostname
[
"v1.4.1”, <- server version
"s50002”, <- SSL port
“t50001” <- TCP port
]
],

There are some commands which are reserved for server to server interaction only. For example, server.add_peer mentioned above is meant to be used by an ElectrumX server to add itself to a peer list of another ElectrumX server.

Crawling the Electrum Network

At this point, I had enough understanding of both the network architecture and the communications protocol to start recursively querying peer lists from ElectrumX servers in order to map out the entire network:

$ python spider.py servers.json --protocol s
963 servers are known to us at start
812 servers are right protocol
165.227.22.180 gave 13 peers
...
found 1 more servers from electrum6086073.livex.biz: electrum666370.tnsfr.link
electrum7583805.rollerco.xyz gave 148 peers
electrum811908.fullhealth.net gave 133 peers
electrum3436028.rollerco.xyz gave 148 peers
electrum8068921.fullhealth.net gave 131 peers
electrum7509195.rollerco.xyz gave 147 peers
electrum6957631.elastics.info gave 150 peers
electrum1231318.tnsfr.link gave 155 peers
electrum1644345.rollerco.xyz gave 149 peers
electrum3912268.tnsfr.link gave 154 peers
electrum4994636.rollerco.xyz gave 148 peers
electrum1014156.tnsfr.link gave 156 peers
electrum345782.tnsfr.link gave 155 peers
electrum2884548.rollerco.xyz gave 148 peers
electrum4699142.tnsfr.link gave 156 peers

As I was slowly aggregating all of the known peers on the network (more than 1000 right now), I started noticing suspicious looking domain names which appeared to be randomly generated and also announced similarly random looking nodes:

$ python cli.py --port 50002 --protocol s server.peers.subscribe --server electrum6086073.livex.biz
Connected to: electrum6086073.livex.biz
Method: server.peers.subscribe
[
...
[
"185.25.48.104",
"electrum5391756.rollerco.xyz",
[
"v1.4",
"s50013",
"t50512"
]
],
[
"185.25.48.104",
"electrum8854308.rollerco.xyz",
[
"v1.4",
"s50020",
"t50519"
]
],
[
"27.102.129.56",
"electrum6001717.livex.biz",
[
"v1.4",
"s50009",
"t50508"
]
],
...

A lot of them had more interesting similarities like being hosted on the same IP address or simply being aliases for the same domain:

$ host electrum5391756.rollerco.xyz
electrum5391756.rollerco.xyz is an alias for rollerco.xyz.
rollerco.xyz has address 185.25.48.104
...
$ host electrum6001717.livex.biz
electrum6001717.livex.biz is an alias for livex.biz.
livex.biz has address 27.102.129.56

Using randomly generated subdomains as aliases to a single domain is a clever exploit of ElectrumX’s peer verification process which simply verifies that the advertised hostname resolves to the originating IP address. As a result, it is possible to flood peer tables with these ghost servers to perform a Sybil attack, where an attacker attempts to fill the network with servers they control to increase their chances to connect with Electrum clients.

Finding evil

But are these servers truly malicious? The best test for whether or not these servers are evil is to trigger the phishing exploit where they would attempt to instruct my “fake” client to open a popup. Based on the notes posted on the Electrum’s GitHub issues page, the error message is forced when a client attempt to broadcast a Bitcoin transaction. I sent a sample bitcoin transaction to the node server using the blockchain.transaction.broadcast verb and indeed received a phishing error message:

Connected to: electrum6001717.livex.biz
Method: blockchain.transaction.broadcast
({'code': 1, 'message': '<b>Transaction error.</b><h1 style="white-space:nowrap">Required Security Update (v3.3.3)</h1><p>Download: <a style="text-decoration: none;" target="_blank" href="https://www[.]electrumbuild[.]com/#download">https://www[.]electrumbuild[.]com</a></p><p>This important security update provides a fix for transaction deserialization vulnerability and is mandatory for all users.</p><p>Transactions can only be sent after applying this update. You can visit the website above to download the new version of Electrum.</p><p>We sincerely apologize for any inconvenience this has caused you.</p><p>© Electrum Technologies GmbH</p>'}, {'id': 2, 'method': 'blockchain.transaction.broadcast', 'params': ('010000000[..ABBREVIATED..]8888ac2c480700',)})

The server attempted to send a crafted html page to whomever was trying to use it, directing them to download a fake update from https://www[.]electrumbuild[.]org.

The phishing website looked almost exactly like the original and even advertised the current version of Electrum Wallet at the time. Of course, all of the binaries were backdoored and neither of the file hashes matched the original.

Finding more evil

After crawling the network for awhile longer, I aggregated almost 1000 unique nodes. It made sense to automate the hunting for malicious servers by scanning a range of Electrum nodes and attempting to trigger a transaction error. All of the error messages were compared against a whitelist of known server responses and any deviations were recorded for further analysis. Below is a sample output of the scanning tool that I wrote for this purpose:

$ python electrohunt.py servers.json --output evil.json
[*] Scanning 967 servers...
[*] Connected to 165.227.22.180:50002 using SSL
[*] Connected to 165.227.202.193:50002 using SSL
[*] Connected to 79.142.76.244:54187 using SSL
...
[*] Connected to 88.198.241.196:50002 using SSL
[*] Connected to electrum3402488.livex.biz:50018 using SSL
[*] Connected to 185.183.158.170:50002 using SSL
[*] Connected to 27.102.129.56:50002 using SSL
[!] electrum3402488.livex.biz:50018 sent an unusual error:
({'code': 1, 'message': '<b>Transaction error.</b><h1 style="white-space:nowrap">Required Security Update (v3.3.3)</h1><p>Download: <a style="text-decoration: none;" target="_blank" href="https://www[.]electrumupgrade[.]org/#download">https://www[.]electrumupgrade[.]org</a></p><p>This important security update provides a fix for transaction deserialization vulnerability and is mandatory for all users.</p><p>Transactions can only be sent after applying this update. You can visit the website above to download the new version of Electrum.</p><p>We sincerely apologize for any inconvenience this has caused you.</p><p>© Electrum Technologies GmbH</p>'}, {'id': 2, 'method': 'blockchain.transaction.broadcast', 'params': ('010000000[..ABBREVIATED..]8888ac2c480700',)})
[*] Connected to 109.192.105.174:50002 using SSL
[!] 27.102.129.56:50002 sent an unusual error:
({'code': 1, 'message': '<b>Transaction error.</b><h1 style="white-space:nowrap">Required Security Update (v3.3.3)</h1><p>Download: <a style="text-decoration: none;" target="_blank" href="https://www[.]electrumupgrade[.]org/#download">https://www[.]electrumupgrade[.]org</a></p><p>This important security update provides a fix for transaction deserialization vulnerability and is mandatory for all users.</p><p>Transactions can only be sent after applying this update. You can visit the website above to download the new version of Electrum.</p><p>We sincerely apologize for any inconvenience this has caused you.</p><p>© Electrum Technologies GmbH</p>'}, {'id': 2, 'method': 'blockchain.transaction.broadcast', 'params': ('010000000[..ABBREVIATED..]8888ac2c480700',)})
...
----------
Successfully connected to 657 nodes.
Failed to connect to 0 nodes.
There were 28 invalid nodes.
----------
Detected 471 potentially evil nodes:
electrum5402863[.]fullhealth[.]net
electrum7969658[.]fullhealth[.]net
electrum9164215[.]fullhealth[.]net

Based on the automated scanning for phishing messages, I observed more than 471 confirmed malicious servers out of a total of 657 active nodes. That’s a staggering 71% of the entire scanned network being controlled by attackers. The vast majority of them appeared to be a part of the same campaign described above. However, there were also a few additional unique attempts at exploitation:

[*] 184.75.221.195:5215 returned an unexpected message:
<b>Error sending transaction.</b>
<h1 style="white-space:nowrap">Security update required (2019-002 v3.3.4)</h1>
<p>This important security update provides a fix for transaction deserialization vulnerability and is recommended for all users.</P>
<p>Transactions can only be sent after applying the update. Please visit the link below to find instructions on how to update to Electrum 3.3.4.</p>
<p><font color='blue'>
https://github.com/e8d1/electrum.org/releases/latest</font><p>
<p>We will post more detailed information in the near future. Please visit our website for more information.<p>

The Github page was created about 5 days earlier and contained several backdoored Electrum “releases”:

The Github security team was very effective at responding to the my takedown request . The attackers have since shut down their ElectrumX servers and appeared to stop their campaign on the Electrum network.

Campaign Analysis

Overall there were at least two ongoing campaigns abusing the Electrum transaction error vulnerability with slightly different tactics.

Campaign #1 relied on a large number of randomly generated and aliased nodes such as electrum5384894[.]lightspeed[.]tel. The campaign advertised a URL with a link to an almost mirror image of the official Electrum website.

Grabbing all of the malware samples from the phishing site, I quickly took them apart to see what attackers modified. The malware samples used in the campaign were based on modified versions of the Electrum-3.3.2 wallet and targeted Windows, Mac, Linux, and Android platforms. A number of modifications to the original were performed to force transactions to go to bc1q92md7868uun8vplp9te0vaecmxyc5rrphdyvxg and later to bc1qhsrl6ywvwx44zycz2tylpexza4xvtqkv6d903q Bitcoin addresses. The backdoored binaries have also disabled Electrum’s update mechanism and other transaction security notifications. It was interesting that the malware was checking for minimum balance of 0.05 BTC before attempting to steal funds.

The actors behind this campaign changed the destination phishing site on almost daily basis. They were actively monitoring and adding new first level domains to avoid detection. As a new version of the official software was released, the actors quickly updated their backdoored Electrum Wallet and the phishing site within a day. I took a look at those as well and they only modified the destination Bitcoin address to steal funds.

The use of aliases to inflate the total number of connected nodes used by the attackers in Campaign #1 was an interesting exploit of the Electrum network’s peer verification system. At one point, they may have controlled up to 71% of the network.

Overall the campaign appeared to be very successful with just one addresses used in the malware aggregating 55 BTC (about $200,000 at the current rate) and almost 10 BTC in another with new transactions showing up hourly. Below is a screenshot listing most recent transactions from Smartbit’s blockchain explorer:

Campaign #2 relied purely on IP addresses with no hostnames for their nodes. This campaign attempted to direct users to a malicious Github page which was also designed to look official. The actors in this campaign advertised wallet version 3.3.4. It appears this group was taking a more stealth approach with fewer but better connected nodes.

I collected two unique collections of malware samples for this campaign. The first collection was targeting Bitcoin users and simply uploaded private keys and seed data to http://4a5e1e4baab[.]com. The second sample was a bit more sophisticated and implemented Litecoin address replacement logic in addition to being able to also upload private keys and seed data to several obfuscated addresses. See Appendix B for more details.

The campaign also appeared to be fairly successful with one of the Litecoin addresses collecting a few thousand dollars worth of LTC:

Additional analysis is necessary to gather threat intelligence on the infrastructure used by the attackers in each campaign to truly understand their scope and actors behind them. However, this will be covered in an upcoming Part 2 of the Electrohunt series.

I listed all of the domains, hashes, and malware indicators observed so far in Appendix A, B, and C below.

Present and future Electrum network threats

After spending some time monitoring the Electrum network, it is clear to me that it is under a persistent attack from one or more determined actors. The current attack pattern is based on exploitation of a known and already patched vulnerability in Electrum Wallets. However, attackers may change their tactics from simply dominating the network to push malicious error messages to attempting to trick Electrum clients’ view of the state of the Bitcoin network in the future.

For example, the attackers could use their dominant position to hide inbound transactions or prevent users from being able to broadcast theirs. There are also privacy risks, as the attacker with the majority of controlled nodes could attempt to deanonymize users by observing all of the IPs associated with a single Bitcoin address.

Electrum Wallet and server developers have recently taken active steps to defend the Electrum network. For example, the latest version of the ElectrumX server implements node blacklists which already captures bad nodes in Appendix C. The Electrum Wallet developers have now completely disabled the vulnerable error popups feature. It is a responsibility of server operators and wallet users to upgrade their software in orders to take advantage of these defenses and to avoid further fund theft.

Remaining vigilant and downloading software only from trusted sources will remain the best advice for users of Electrum Wallet.

The field of cryptocurrency security is still very young. I hope some of you readers will be motivated to join me in this new field and contribute to making the future open financial system more trusted and more secure. Check out our open careers at coinbase.com/careers.

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

Unless otherwise noted, all images provided herein are by Coinbase.

References

Appendix A — Discovered Phishing Sites

Campaign #1

https://www[.]electrumupgrade[.]org
https://www[.]electrumclient[.]org
https://www[.]downloadelectrum[.]org
https://www[.]electrumsite[.]com
https://www[.]electrumweb[.]net
https://www[.]electrumupdate[.]com

Campaign #2

https://github[.]com/e8d1/electrum.org/releases/latest

Appendix B — Malware

Campaign #1

ec0d6d90c0c63ebc635280ccf481c374  Electrum-3.3.3.0-release.apk
f53d2f76d36932bbf3f7762b18b7b278 electrum-3.3.3.dmg
b53905bef4194fd247511d1788ea36f8 electrum-3.3.3.exe
590ed8124a453acd65f753c0eae1c54f electrum-3.3.3.exe.asc
48bab01fd9d1f64039a792aaf7740492 electrum-3.3.3-setup.exe
1460670adf6184b92005fe47f0c3f519 electrum-3.3.3-setup.exe.asc
49ae3d041ef9734db7c09395dca10dc9 Electrum-3.3.3.tar.gz
55db5df25aa283c61050043da29ca706 Electrum-3.3.3.zip
Bitcoin address: 
bc1q92md7868uun8vplp9te0vaecmxyc5rrphdyvxg
ec0d6d90c0c63ebc635280ccf481c374  Electrum-4.0.0.0-release.apk
cda16f12ecac49163ddb0f36bca579e8 electrum-4.0.0.dmg
dd435f1bc9bbfcd6abdc9412a5f076c4 electrum-4.0.0.exe
8b2fd24ccd23aa749d4847d2e1373699 Electrum-4.0.0.tar.gz
3122fa940d25911853436497b4821e4b Electrum-4.0.0.tar.gz.asc
c196c792629dd73e4f1f1da724367b2b Electrum-4.0.0.zip
Bitcoin address:
bc1qhsrl6ywvwx44zycz2tylpexza4xvtqkv6d903q

Campaign #2

677203ab5ad2b8116893fcbd71d497ce  Electrum-3.2.3.1-release.apk
926d4a2dcf93d2012456d5f0058cd62b electrum-3.3.4.dmg
1a2509b48230b90b64a2b10625bce801 electrum-3.3.4.exe
dbb13365c1cfc5df5f480670827b6a36 electrum-3.3.4-portable.exe
bf6e1be858bfa07f8d625373ff3ab80d electrum-3.3.4-setup.exe
809c9faa38cbfa3deeca012bee7e3275 Electrum-3.3.4.zip
Payload dump site: http://4a5e1e4baab[.]com
POST request with “data=” parameter
b2dd8dc636909577fdaf57ad728c677e  electrum-ltc-3.3.4.dmg
a8142ae8fa735414a8b435f0912dea0d electrum-ltc-3.3.4.exe
58ee1fcf0d2245b8e8b3094dd24aa6a6 electrum-ltc-3.3.4-portable.exe
09de0a6c2149edfb4093b4978afd364f electrum-ltc-3.3.4-setup.exe
6eda4fc5243586c42f1d8727c83d2fc3 litecoin-3.3.7.zip
Payload dump sites:
https://hcegdqqm[.]club:57897
https://ykmkrgfkc[.]ru:57897
https://194.36.111[.]59:57897
https://193.37.254[.]35:57897
https://185.104.184[.]43:57897
Litecoin addresses: 
LhbGVfuLPxMMFZrkDkSPC16Yu2RQ7S9Bc7
M8wRkiNmi5K5339LE8ncpWLYgbWmR121JL
Le3gXVa4SshHs3TmdrWNC434ceDQtsjMt8

Appendix C — Malicious Electrum Nodes

Campaign #1

NOTE: replace nodes with [\d*] with randomly generated digits (e.g electrum6313288.arcade[.]tel or simply electrum.arcade[.]tel)
46.166.160.164
111.90.15.213
185.200.190.209
185.25.48.104
185.25.48.36
185.8.177.126
27.102.129.56
85.206.160.91
electrum[\d*].arcade[.]tel
electrum[\d*].zzfis[.]xyz
electrum[\d*].btcx[.]link,
electrum[\d*].bip[.]click,
electrum[\d*].elastics[.]info'
electrum[\d*].esrv[.]one',
electrum[\d*].fullhealth[.]net'
electrum[\d*].lightspeed[.]tel
electrum[\d*].livex[.]biz'
electrum[\d*].rollerco[.]xyz'
electrum[\d*].ssrv[.]info
electrum[\d*].tnsfr[.]link'
electrum[\d*].txid[.]pw'
electrum[\d*].xs500[.]net'
electrum[\d*].electrumxm[.]com
alpha.zzfis[.]xyz
beta.zzfis[.]xyz
electrumx.zzfis[.]xyz
fn.zzfis[.]xyz
gamma.zzfis[.]xyz
host.zzfis[.]xyz
io.zzfis[.]xyz
oneweek.duckdns[.]org
ray.zzfis[.]xyz
zzfis[.]xyz

Campaign #2

184.75.223.219
185.9.19.107
184.75.223.203
184.75.221.59
184.75.221.195
194.99.104.35
199.241.147.35
213.152.180.5
217.151.98.168

Electrohunt Part 1: Hunting for the phishing campaigns on the Electrum network was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more on Coinbase
Posted on February 15, 2019
Earn Zcash while learning about blockchain privacy

We’re launching a new Coinbase Earn page where users can earn Zcash (ZEC)

Today we’re launching a new Coinbase Earn opportunity with lessons about Zcash (ZEC).

If you’ve received an invite to the Coinbase Earn ZEC Page by email, you can go to the Earn ZEC page to start completing the educational lessons and earn up to $3 worth of ZEC in the process.

Zcash is a cryptocurrency which uses recent advances in cryptography to give users the option to protect their financial privacy when transacting with cryptocurrency. Zcash does this by offering two types of addresses–“transparent” addresses and “shielded” addresses. A good analogy for this distinction is HTTP (unencrypted) and HTTPS (encrypted). In both cases, the unencrypted version of the protocol is transparent, which allows third parties to see information about the transaction, while the encrypted/shielded version protects this information. This can help keep your financial information safe, so it can’t be sold, shared, or spied on by third parties. You can learn more about Zcash and its privacy design by watching the educational videos on the Earn ZEC Page.

We expect earning to become an increasingly important function in the crypto ecosystem–alongside buying and mining–especially when paired with education. We hear from people that they’re always looking for trusted sources to educate themselves about new developments in crypto, and we want to help serve that need.

If you haven’t received an invite to earn yet, don’t worry — you can still view all content on the page for free, and you can join the waitlist to be notified once earning opportunities become available to you. If you want to stay informed about future opportunities, please make sure you have verified your identity and opted in to marketing communications with Coinbase!

Lastly, you can check out the newly-launched Coinbase Earn homepage to keep up to date with the latest Earn opportunities.


Earn Zcash while learning about blockchain privacy was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more on Coinbase
Posted on February 12, 2019
Backup your private keys on Google Drive and iCloud with Coinbase Wallet

Backup your encrypted private keys on Google Drive and iCloud with Coinbase Wallet

Starting today, you can now backup an encrypted version of your Coinbase Wallet’s private keys to your personal cloud storage accounts, using either Google Drive or iCloud.

This new feature provides a safeguard for users, helping them avoid losing their funds if they lose their device or misplace their private keys.

Explore the open financial system with peace of mind

Since we launched Coinbase Wallet (formerly Toshi), our users have experienced the full power of an open financial system — storing their own funds and accessing them anywhere in the world.

But with that power comes great responsibility. The private keys generated and stored on your mobile device are the only way to access your funds on the blockchain. Owners of ‘user-controlled wallets’ like Coinbase Wallet sometimes lose their devices or fail to backup their 12 word recovery phrase in a safe place, thus losing their funds forever.

Now, with cloud backup, we give you the ability to store an encrypted copy of your recovery phrase on your personal cloud account. You will only have to remember a password, that you decide, in order to recover your funds. If you lose your device or get signed out of the app, you can easily regain access to your funds with the combination of your personal cloud account (iCloud or Google Drive) and your password.

Your backup is encrypted with AES-256-GCM encryption and accessible only by the Coinbase Wallet mobile app. The backup can only be decrypted using your password.*

Coinbase will not have access to your password or funds at any time, preserving your privacy and control. Your cloud backup provider will also not have access to your funds, as only you know the password that decrypts your encrypted recovery phrase.

The cloud backup feature currently supports iCloud on iOS devices and Google Drive on Android devices. We intend to add support for other cloud services in the future.

Cloud backup is optional and you have to opt-in to activate it. You will still have the option to view your recovery phrase and back it up manually as before. Because we offer cloud backup as a convenience, we recommend that users also backup their passphrase manually after activating the cloud backup service. For additional security, remember to activate Two-Factor Authentication on your personal Google or iCloud accounts to make those accounts harder for attackers to compromise.

How to activate cloud backup

When you update your Coinbase Wallet app to the latest version in the next few days, you will start to receive notifications to backup your private key to the cloud.

You can also enroll in cloud backup at any time from the Settings menu, by tapping on ‘Recovery Phrase’ and following the prompts.

We’re also introducing periodic notifications to remind you to backup your recovery phrase to the cloud or manually if you haven’t done so already.

Restoring your wallet

To restore your wallet on a new device or from a logged out state, just tap ‘I already have a Wallet’, enter the password you used to backup, and voila, your account is recovered! Note that your phone must be logged into the same Google account (Android) or iCloud account (iOS) that you originally used to backup your wallet.

Finally, if you juggle multiple wallets, no problem! Our cloud backup service supports multiple wallets so you can easily switch between accounts.

If you have only ever used the primary Coinbase app (or Coinbase.com), we encourage you to check out Coinbase Wallet. The Coinbase Wallet app is Coinbase’s user-controlled cryptocurrency wallet. With Coinbase.com, you can buy crypto and Coinbase stores it (along with your private keys) for you; with Coinbase Wallet, you store your own crypto (safeguarded by a private key that only you know). To learn more, visit our website.

We hope you enjoy using Coinbase Wallet. Your feedback helps us make Wallet better for everyone. You can reach us at wallet.support@coinbase.com.

Unless otherwise noted, all images provided herein are by Coinbase.

*Updated to include details on backup encryption.


Backup your private keys on Google Drive and iCloud with Coinbase Wallet was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more on Coinbase
Posted on February 11, 2019
Logs, metrics, and the evolution of observability at Coinbase

Up until mid 2018, the only supported way to generate custom metrics for an application at Coinbase was to print to standard output. Each printed line would work its way through a series of pipelines where they could be analyzed, transformed, and parsed until finally landing on an index in our monolithic self-managed Elasticsearch cluster, where engineers could build visualizations and dashboards with Kibana.

This workflow represents the best of many worlds. Log entries are easy to generate, and because they can contain fields with unlimited cardinality at microsecond granularity, it’s possible to slice, dice, and aggregate these entries to diagnose the root cause of even the most complex issues.

In 2016, we reaffirmed our decision to self-manage Elasticsearch. At the time, managed solutions like Amazon Elasticsearch Service were still in their infancy (for example, IAM policy changes would require a full redeploy of the cluster) and we were reluctant to trust not yet mature third party vendor technology at the time with the potentially sensitive nature of our applications’ standard output data. We built elaborate automation to manage and blue-green deploy our Elasticsearch data, master and coordinating nodes with our existing deployment pipeline and comply with our self-enforced 30 day project.

Visualizing our elaborate Elasticsearch deployment strategy.

Managing this newly automated Elasticsearch ourselves actually worked out fine for over a year. As an organization of thirty engineers serving up two moderately trafficked web applications, both our indexing (~100GB per day) and search volume (a few queries per minute) were relatively tame.

Backend API requests per minute over the course of 2017.

However during 2017, interest in cryptocurrency skyrocketed, catapulting our moderately trafficked web applications into the spotlight. This massive surge in traffic to our customer facing applications (60x!) had the downstream effect of an equivalent surge in downstream log entries. Earlier technical challenges contributed to this surge of new entries as engineers worked overtime to add diagnostics to our platform (see our previous scaling blog post for more detail). During this period, a single Coinbase API call could generate up to 100 separate log entries, at times resulting in terabytes of log data per hour.

The cluster became unwieldy. As the size of the engineering team increased dramatically, the combined weight of terabytes per day of logs, a monolithic cluster, and unrestricted access patterns led to outages that halted engineering productivity and created operational hurdles.

The logs experience for engineers during Elasticsearch outages.

It became clear that a monolithic Elasticsearch cluster would not be sufficient for our observability needs. Some of our problems included:

  • A single bad Kibana query (hello, Timelion) could cause a hiccup severe enough to affect the entire engineering organization.
  • Certain issues were difficult to diagnose or prevent — Elasticsearch, Kibana and X-Pack lack the ability to analyze slow/expensive queries by user nor the controls to prevent expensive queries like multi-week aggregations.
  • Many of Elasticsearch’s load-related failure conditions (like a series of very expensive queries) would leave the cluster in a disabled state even after load had entirely subsided. Our only recourse to recover from these crippled states was to perform a full cluster restart.
  • Elasticsearch does not have a separate management interface, meaning that during failures we would be unable to perform basic diagnostics on the cluster. Opening a support request with Elastic requires diagnostic dumps that we were unable to generate until the cluster had recovered.
  • Alerting on data stored in Elasticsearch data was not intuitive — tools like Elastalert and Watcher don’t allow engineers to interactively build alerts and were difficult to integrate with PagerDuty and Slack.
  • We were forced to reduce log retention to reduce the impact of massive queries and speed up cluster recovery following failures.
  • Aside from application server standard output, we were also storing and querying security data like AWS VPC Flow Logs and Linux Auditd records which required separate security controls and exhibited different performance characteristics.
  • Our two largest internal applications consumed 80–90% of our log pipeline capacity, reducing the performance and retention for other smaller applications in our environment.

We chose to solve these problems in two ways:

  1. Introduce a new platform to provide engineers with features that are impractical to provide with Elasticsearch (namely inexpensive aggregations, alerting, and downsampling for long retention).
  2. Split our self-managed, monolithic Elasticsearch cluster into several managed clusters segmented by business unit and use case.

Metrics

Based on our challenges self-managing Elasticsearch combined with our short experiments at developing a reliable blue-green deploy strategy for Prometheus backends influenced our decision to choose a managed service for our metrics provider. After evaluating several providers, we settled on Datadog.

Unlike log platforms which store data as discrete structured events, metric stores like Datadog store numerical values over time in a matrix of time by tag/metric value. In other words, in Datadog, if you send a single gauge or counter metric 100,000 times in a 10s period, it will only be stored once per tag value combination while on a logs platform that same value would result in 100,000 separate documents.

As a result of this tradeoff, metrics stores like Datadog allow for extremely fast aggregations and long retention at the cost of reduced granularity and cardinality. Specifically with cardinality, because every additional tag value added to a given metric requires a new column to be added to the time vs tag/metric matrix, Datadog makes it expensive to breach their 10k unique tag combination per host limit.

Despite these tradeoffs, we’ve found Datadog to be a near-perfect complement to our internal logs pipeline.

  • Creating and iterating on alerts is fast and intuitive. Internally, we’ve built a tool to automatically monitor and codify our Datadog monitors and dashboards to prevent accidental modifications.
  • Pulling up complex dashboards with metrics from the past several months is guilt-free and near-instantaneous.
  • Features like distribution metrics allow engineers to easily understand the performance from a block of code with global percentiles.
  • Limitations on cardinality can be confusing for engineers who want to keep track of high cardinality values like user_id or wallet address.

Metrics vs Logs

A common topic internally, we’ve found the distinction between Metrics and Logs to be important. At a high level the tools have similar feature sets — both allow applications to send arbitrary data which can be used to combine visualizations into fancy dashboards. However, beyond their basic feature sets, there are major differences in the features, performance, and retention of these tools. Neither tool can support 100% of use cases, so it’s likely that engineers will need to leverage both in order to provide full visibility into their applications.

In short, we think that Metrics should be used for dashboards and alerts; while Logs should be used as an investigative tool to find the root cause of issues.

An table we use internally to illustrate the difference between Logs and Metrics.

Securing the Datadog Agent

From a security perspective, we are comfortable using a third-party service for metrics, but not for logs. This is because metrics tend to be numeric values associated with string keys and tags, compared to logs which contain entire lines of standard output.

Datadog offers operators a series of tight integrations — if you provide the Datadog agent access to a host’s /proc directory, Docker socket, and AWS EC2 metadata service, you’ll be provided rich metadata and system stats attached to every metric you generate. Running a 3rd party agent like Datadog on every host in your infrastructure, however, carries some security risk regardless of vendor or product, so we chose to take a more secure approach to employing this technology.

We took several actions in order to gain maximum exposure to these Datadog integrations while also reducing the risk associated with running a third party agent.

  • Rather than use the pre-built Docker container, we built our own stripped down version with as few optional third party integrations as possible.
  • By default, the agent is opt-in — projects need to explicitly choose to allow the Datadog container.
  • On every host, put the container on a separate “untrusted” network bridge without access to other containers on the host, our VPC, or the EC2 AWS metadata service. We hard code the `DD_HOSTNAME` environment to the host’s instance ID to allow the AWS integration to continue working.
  • Run a special Docker socket proxy service on hosts to enable the Datadog container integration without exposing the container’s potentially secret environment variable values.
An overview of how we lock down the local Datadog agent.

Logs

Developing a strong metrics foundation at Coinbase helped to alleviate some of the problems we experienced with Elasticsearch as many workloads naturally migrated to Datadog. Now at least when an issue did occur on the monolithic cluster, engineers had data they could fall back on.

But the Elasticsearch issues and outages continued. As engineer headcount continued to grow, logging continued to be a source of frustration. There were about 7 new engineering productivity impeding incidents in Q4 2018. Each incident would require operational engineers to step through elaborate runbooks to shut down dependent services, fully restart the cluster, and backfill data once the cluster had stabilized.

The root cause of each incident was opaque — could it be a large aggregation query by an engineer? A security service gone rouge? However the source of our frustration was clear — we’d jammed so many use cases into this single Elasticsearch cluster that operating and diagnosing the cluster had become a nightmare. We needed to separate our workloads in order to speed incident diagnosis and reduce the impact of failures when they did occur.

Functionally sharding the cluster by use case seemed like a great next step. We just needed to make a decision between investing further in the elaborate automation we’d put in place to manage our existing cluster, or reapproach a managed solution to handle our log data.

So we chose to reevaluate managed solutions for handling our log data. While we’d previously decided against using Amazon Elasticsearch Service due to what we considered at the time to be a limited feature set and stories of questionable reliability, we found ourselves intrigued by its simplicity, approved vendor status, and AWS ecosystem integration.

We used our existing codification framework to launch several new clusters. Since we leverage AWS Kinesis consumers to write log entries to Elasticsearch, simply launching duplicate consumers pointed at the newly launched clusters allowed us to quickly evaluate the performance of Amazon Elasticsearch Service against our most heavy workloads.

Our evaluation of Amazon Elasticsearch Service went smoothly, indicating that the product had matured significantly over the past two years. Compared to our previous evaluation, we were happy to see the addition of instance storage, the support of modern versions of Elasticsearch (only a minor version or two behind at most), as well as various other small improvements like instant IAM policy modification.

While our monolithic cluster relied heavily on X-Pack to provide authentication and permissions for Kibana. Amazon Elasticsearch Service relies on IAM to handle permissions at a very coarse level (no document or index level permissions here!). We were able to work around this lack of granularity by dividing the monolith into seven new clusters, four for the vanilla logs use case, and three for our various security team use cases. Each cluster’s access is controlled by leveraging a cleverly configured nginx proxy and our existing internal SSO service.

“Kibana Selector” — our way of directing engineers to the appropriate functionally sharded cluster.

Migrating a team of over 200 engineers from a single, simple to find Kibana instance (kibana.cb-internal.fakenet) to several separate Kibana instances (one for each of our workloads) presented a usability challenge. Our solution is to point a new wildcard domain (*.kibana.cb-internal.fakenet) at our nginx proxy, and use a project’s Github organization to direct engineers the appropriate Kibana instance. This way we can point several smaller organizations at the same Elasticsearch cluster with the option to split them out as their usage grows.

Our current log pipeline architecture overview.

Functionally sharding Elasticsearch has not only had a massive impact on the reliability of our logging pipeline, but has dramatically reduced the cognitive overhead required by the team to manage the system. In the end, we’re thrilled to hand over the task of managing runbooks, tooling, and a fickle Elasticsearch cluster to AWS so that we can focus on building the next generation of observability tooling at Coinbase.

Today, we’re focusing on building the next generation of observability at Coinbase — if these types of challenges sound interesting to you, we’re hiring Reliability Engineers in San Francisco and Chicago (see our talk at re:invent about the types of problems the Reliability team is solving at Coinbase). We have many other positions available in San Francisco, Chicago, New York, and London — visit our careers page at http://coinbase.com/careers to see if any positions spark your interest!

This website may contain links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

Unless otherwise noted, all images provided herein are by Coinbase.


Logs, metrics, and the evolution of observability at Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more on Coinbase
Posted on February 6, 2019
Earn BAT while trying out the blockchain-friendly Brave browser

We’re launching a new Coinbase Earn page where you can earn Basic Attention Tokens (BAT)

Today we’re launching a new Coinbase Earn opportunity with lessons about BAT, the Basic Attention Token.

If you’ve received an invite to the Coinbase Earn BAT opportunity by email, you can go to the Earn BAT Page to start completing the educational lessons and earn up to $10 worth of BAT in the process.

Basic Attention Token seeks to improve the efficiency of digital advertising with a token that can be exchanged between publishers, advertisers, and users on the Ethereum blockchain. The token can be used to obtain a variety of advertising and attention-based services on the BAT platform and the Brave browser.

Along with the series of educational videos where you can learn more about the Brave browser and BAT, customers will also have the option to earn BAT by using the Brave browser. We hope this will encourage people to explore and use utility tokens first-hand in their native environments, like the Brave browser.

We see these interactive, advanced lessons in cryptocurrencies and tokens as an important step toward building awareness and usage of utility use-cases in the crypto ecosystem–one where a beginner can quickly get up to speed on a token in a few minutes, actually engage with the product, and earn some of that token along the way. We think learning about crypto and earning crypto is a great way to get people to evaluate and use more utility tokens without having to buy them outright.

We expect earning to become an increasingly important function in the crypto ecosystem– alongside buying and mining– especially when paired with education. We hear from people that they’re always looking for trusted sources to educate themselves about new developments in crypto, and we want to help serve that need.

If you haven’t received an invite yet, don’t worry — you can still view all content on the Coinbase Earn BAT page for free, and you can join the waitlist to be notified once earning opportunities become available to you. If you want to stay informed about future opportunities, please make sure you have verified your identity and opted in to marketing communications with Coinbase!

Lastly, you can check out the newly launched Coinbase Earn homepage to keep up to date with the latest Earn opportunities.


Earn BAT while trying out the blockchain-friendly Brave browser was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Read more on Coinbase

See all news of Coinbase

Coinbase Comments:

Add your comment