Disable Chainlink limits for price oracles in crvUSD

Currently, all price oracles in crvUSD are having +/- 1.5% “safety limits” from Chainlink. It means, if Chainlink price deviates from internal oracle prices (EMA) too much (by 1.5%), it will use the source from Chainlink.

It appears that when market is very volatile, Chainlink (or, rather, market spot) prices cause more losses than necessary. And if the fluctuation is high enough, price oracle which uses chainlink limits can still deviate:
ema-price

As such, I propose to disable Chainlink limits in price oracle contracts for all markets except the old frxeth one (which is to be replaced with a new one shortly). This is done by calling set_use_chainlink(False) for all the price oracle contracts from the ControllerFactory owner (Ownership DAO).

8 Likes

Graph of losses which correlates with the "chainlink limit spikes) around blocks 300 and 650
Figure_1

5 Likes

Thanks Mich. Appreciate you bringing this up.

Which of the two have been more accurate between Chainlink and Curve’s oracles? EMA lines obviously are moving averages and so do not take into account the exact volatility you’re describing, so it is hard to tell from this graph what is accurate.

Crypto markets are volatile, that is probably not going away any time soon. These kinds of losses would occur regardless during price deviation spikes, it is not (or should not be) the job of the oracle to limit losses by erroneous price fluctuation tracking. This would open up potentially malicious arbitrage opportunity and worse, possible attack vectors that would be far more devastating than the losses you’ve outlined here.

If the reason for the discrepancy is that the Curve oracles are not actually tracking the volatility spikes accurately then why not just utilize Chainlink oracles? What is the purpose of the internal oracle prices exactly?

Or, if the internal oracle prices are always the more accurate ones, why are we utilizing Chainlink?

Which is using more sources? I can see the CRV/USD Chainlink oracle uses 19 sources, how many do the internal oracles use?

4 Likes

Really enjoy Mich’s graphs.

Yes, I suppose with V-shaped price movements, Chainlink could incur more losses than just using internal EMA. But what if there is no rebound at all after the drop, or just a small rebound compared with the drop?

Maybe you are betting the most possible price movement patterns and choose an oracle plan fitting that the best?

2 Likes

Trick is that more accurate graphs in case of LLAMMA are WORSE. You need to smoothen things (and that is what simulations were done with)

2 Likes

If there is a stepwise drop and no rebound - yes loss will happen. But tldr: ema is better than a real-time oracle lagging by 1 blocktime.

No rebound whatsoever is no cool indeed, so that is what soft liquidation parameters are chosen from - historic data for all price fluctuations over several years. And EMA time is chosen to work well with it

4 Likes

19 Chainlink nodes does not equal 19 data providers. They could all be calling Coingecko for what we know.

Noo it’s not gecko. It’s actually better than that, and they do match spot prices

You cannot prove who their data sources are, other than a ‘trust me bro’ from the team. They are a 3rd party oracle. They have 19 node operators who all source their data from whoever (and often the same source).

These middlemen are aggregated, not unique data sources.

1 Like

Little update.

Dynamic fees make losses really really small if one spike in price oracle is created, and then EMA makes it decay.

Basically, if 4 (!) blocks in a row get taken over by a malicious MEV manipulator to pump the price oracle, loss comes down to 0.6% at N=50 (while Chainlink-induced loss was 1.5% at maximum!). This manipulation is not feasible (even producing 3 blocks in a row by manipulator is a bit questionable). To make it worse for the attacker - dynamic fee works in such a way that profit from this manipulation (which is expensive) will be taken NOT by the manipulator.
manipulation-loss

Conclusion: in this particular case, EMA oracles look safer than even non-manipulatable unfiltered spot prices

4 Likes

There is another indicator which may correlate with losses, i.e., arbitrageurs’ profits ( including gas and that paid to block builders).

I extracted some 100K U + arbitrages involving LLAMMA during the price drop on Aug 17, and some 100K U + AAVE liquidations as well.

LLAMMA case 1: 500E volume & 1% profit
LLAMMA case 2: 1200E volume & 2.6% profit
LLAMMA case 3: 1200E volume & 2.8% profit

AAVE case 1: 146E volume & 3.5% profit
AAVE case 2: 2426E volume & 5.6% profit
AAVE case 3: 156E volume & 4.5% profit

Looks LLAMMA provides opportunities of smaller profits than AAVE, which is good.(?) But given AAVE’s ‘dumb’ fixed liquidation bonus rate, should we expect LLAMMA performs even better than that?

Maybe because of chainlink oracle, LLAMMA arbitrageurs got more profits? Maybe also because arbitrageurs are still getting familiar with LLAMMA, not too much competition there as in AAVE liquidations?

Mich mentioned soft liquidation parameters chosen based on historical data. I suppose f/g in the LLAMMA invariant are also part of them.

Basically different choices of f/g determine how faster LLAMMA price ( of a specific band ) deviates from Po when Po deviates from the band range, thus determine how soon arbitrageurs step in? And I suppose the answer can not be that simple as ‘sooner better’.

How is this related to losses? Seems quite complex dynamics behind.

Could Mich kindly share some thoughts on how you chose the f/g? Maybe a little bit off topic, but wanna take this opportunity.

2 Likes

This might be a minor concern, but I could imagine removing the “safety limit” opens up a potentially profitable arbitrage for crvUSD borrowers - which I assume is why the limit was created in the first place? Using some dummy numbers:

ETH price $1000
EMA price $1200
LTV: 90%

Arb: deposit 1 ETH, withdraw 1080 crvUSD, profit 80 crvUSD relative to the market prices. If we take an unbiased view on market prices, then LLAMMA just ate a loss, although it might be recouped if ETH prices revert.

Perhaps it makes sense to raise the safety limit to prevent this arbitrage from happening (e.g., a function of LTV at 4 bands). This might be a minor concern for lower volatility assets like ETH but could be a bigger concern for higher volatility assets, if those are ever onboarded to LLAMMA.

Could the team share thoughts on why 1.5% safety limits were chosen, and/or if raising the safety limits to ~10% is a viable option? Thanks!

4 Likes

Good point.

So seems the higher the LTV, the higher the vulnerability of the protocol because of the oracle.

1 Like

Actually these particular cases you mentioned ARE having relatively high losses due to using Chainlink! So this is exactly my concern

3 Likes

Now as we have this off - let’s observe how it behaves in similar volatility

2 Likes