Connect Working Group session
RIPE 80
13 May 2020
1 p.m.
REMCO VAN MOOK: All right. Good afternoon everyone and welcome to the virtual Connect Working Group session on a virtual RIPE meeting on an actual Wednesday afternoon. I am Remco van Mook, I am co‑chair of this Working Group together with Will Van Gulik, who is here, and Florence, who, due to work obligations, will not be able to join us today.
Now, to get started with this. I had to, for this virtual session, I had to go through my stack of technology to find something to do the slides with and it was a little bit more complicated than I wanted it to be. So I found this overhead projector, so I'm hoping this is going to work, so let me pull up the first slide.
So, this is the agenda for the session. Let's see what we have.
There is the opening. I guess that's this. The scribe, I think that's already been announced in the chat.
The agenda, the format. We're going to try to have a couple of discussions during this session, during the challenge of operations under Covid‑19.
Then there is a coffee break. And then after that, some more, there is a bit of an open discussion on what's going on with the traffic statistics and what is the answer we'll see of the Internet and how true is that still?
There is Raffaele, who is going to present, unfortunately didn't make it to this slide. Then we have a Covid‑19 update from EURO‑IX peering tool box update from Nurani. I missed another presentation, Massimo, is going to be presenting in the second time slot as well so I'll update my slide, I am going to have a write a new one. And that's it.
Any current ‑‑ let's see if there is anything in the chat or the Q&A yet?
I found a chat window as well. Let's see, I'm not going to wear a shark on my head. Okay. I guess that nobody disagrees.
So, then I would like to draw your attention to the minutes of the previous session. I assume all of you have be reviewed them at length and are happy with them, and if I don't hear and see anything within the next ten seconds, I am going to assume that you are all in favour of approving them.
Okay. That's it.
So, let's see what else is there? Some of the standard stuff.
If you have questions for presenters, use the Q&A button in Zoom only or use the IRC channel, and the RIPE NCC monitors will then relay your questions.
There is that. Let's see. Oh yes! This Zoom chat has one emergency exit, it is the bright red button on the bottom right hand of your screen, and if you are watching this on the webcast on a web browser, then just very calmly close the window.
With that, I think that was everything I needed to say. Let's go to Will, I think?
WILL VAN GULIK: Let me find my things. That's the one I need to share.
Obviously we're at Connect, we're doing things correctly, you know. So, welcome.
I will present you about a small update on what's going on recently with some funny and interesting peering things in Switzerland. It's a court case from ‑‑ here is a small agenda of what I will cover. A small Whoami disclaimer. I'll tell you about the back story of that, and then what's the status of this justice decision and what it will lead to and then maybe we'll have a bit of time for questions.
So I am this guy, you see me there. That's me. I do things obviously with the Internet hopefully that are useful because it seems that people are able to watch the stream correctly. So, disclaimer:
I do not represent any of the involved parties. I am just a messenger, so don't shoot me. The opinion expressed in this are my own and not necessarily of those of my employer or any affiliation. Obviously I'm not a lawyer and I'm not even a German speaker, so, because the judgments of that case was totally in German, it was a bit hard for me to grasp all of it, so, caution may be advised.
So, rough summary:
On April 29th, Fredy Kuenzler mentioned on the NANOG mailing list that Init 7 won a court case against Swisscom on the topic of peering.
The back story is, in 2011, like that's a long time ago in a galaxy far, far away, Init established and Swisscom established a zero settlement peering session. This agreement was ended in 2012 by Swisscom who wanted to suggest a new proposal that would imply charges for the traffic ratio if it exceeds 1: 2.
And so, in 2013, Init 7 filled a request to the ComCom, which is like basically the Swiss FCC, to obtain a free peering settlement to Swisscom, to access the Swisscom network and that's actually where the legal part started. Like figuring that will trigger a court case and therefore, because the case was a bit complex, the ComCom granted that, they enforced Swisscom to go and set up the peering session.
So, also, worth to be mentioned. So at the time, Swisscom had an agreement with Deutsche Telekom that states that a person ‑‑ so Swisscom traffic that goes to the Internet, must be sent to the DTAG. Swisscom got some part of Deutsche Telekom transit benefit when DTAG sends traffic back to Swisscom. I think that makes sense.
Then the interesting part is that it obviously could have been insisted for Swisscom to avoid any free traffic change and do something like, like do some paid peering type of agreement because, like, it doesn't cost anything to them, and they actually generate some benefits.
It's worth noting that this agreement was ‑‑ ended in 2016. And so, if you want to go in into the specific details, that's all available in the legal document which is available online, I got also a link at the end of my slides, and this is a 42‑page German legal document. So, it's a bit, let's say, heavy.
The justice decision. What happened there:
So we have to understand that we are at the highest level of the justice in Switzerland, so basically you have got the Supreme Court, which has three layers. One of them is the Federal administrative courts, the other is like the financial, I cannot remember the other one. So, basically we're on a non‑irrevocable decision, so, Swisscom ‑‑ so what they noticed is that Swisscom was indeed in a dominant position until the agreements with DTAG ended. It also means that the ComCo, which is not the same as the ComCom obviously, the Competition Commission, so the ComCo must evaluate what would be a price for the service that should be applied for this period that goes up to the end of 2016.
And obviously for the period after 2016, which runs up to now when the judgement was given back, it's the lower‑instance tribunal, so I guess it's the local tribunal, that must clarify what's the situation for this agreement.
So, one other thing to note as well, like the Init 7 did put some money upfront to go deal with that court case. Those have been waived and there is an indemnity that will be given by Swisscom to Init 7.
So, what now?
We have got more waiting to do, obviously, because now we have got the ComCo that will give the price and the ComCo that the ‑‑ the lower‑instance tribunal who will also state on the other timing.
So, now, we have got some several interesting questions, is like: Will this affect the prices that Swisscom applies for a paid peering service even though this case covers up to 2016? So maybe the situation is different right now. So that's a good supposition, we can ask that.
We were wondering if that will apply to other ISP? I mean big ones, Swisscom is obviously the incumbent, but we have also got some fairly good and big ISPs in Switzerland, so maybe that could apply there.
And the other question is like, maybe that will be an incentive or a question for other countries that will want to state something like that. And I think we noticed some interesting case like that in France recently, obviously with the Covid situation, where Orange, France Telecom, gave a bit of slack on their peering agreement. Obviously now we are waiting for feedback from maybe the parties on that case. It's probably at the next RIPE. So I will be waiting for that.
So, basically, that's it from me. I just want to make a short big thanks to sieve yen on that one, because he really gave me a really a lot of help with the explanations and so on. So that was really really nice. And all these are my sources, you can also get to that document.
And now I think we can go and see if we have any questions?
REMCO VAN MOOK: Thank you very much for that, Will. I see two questions in the Q&A so far.
I have a question. So, we're now basically waiting for the Swiss regulator to set a tariff on peering, is that correct?
WILL VAN GULIK: Exactly. At least for the peering with Swisscom. That's also on the period that goes up to 2016, which is an interesting part, because now that's ‑‑ Swisscom doesn't have that agreement with DTAG any more, the situation is a bit different. So, maybe that will affect the situation and that's what we are waiting for the lower‑instance tribunal, and so that's an interesting situation.
REMCO VAN MOOK: How is that going to impact basically all peering in Switzerland going forward? Because if there is regulated tariff, I can see how that would be applied to everyone and everything or ‑‑ is there any language in the verdict that says anything about that?
WILL VAN GULIK: I didn't spot anything like that. The thing is that most of the other people in Switzerland do eventually free peering, like, so we just go and connect and with our IXPs just connect to the XP and then you peer. I would think, if you want to put a fee on the access to your network, then that's the specific case where it would occur. But I don't see anyone willing to do that besides, I mean, Swisscom and maybe, don't take my word for it, but UPC.
REMCO VAN MOOK: Okay. I have a number of questions. First one is from Marco d'Itri from Seeweb:
"I remember that Swisscom is over 50% of the access market. Why did the regulator not affirm that this is a dominant position even after 2016?"
WILL VAN GULIK: That's an interesting question. I think, Marco, you should go and read the legal document. Like, it's a bit beyond my knowledge for legal stuff. It's a really good question. I was wondering that, and it didn't appear really anywhere. So...
REMCO VAN MOOK: Maybe it's good to, if you can compile a couple of links and send it out to the Working Group mailing list, so people can read up on this. I think that this has potential repercussions all over the RIPE regions if this really happens. So I think it would be good for a lot of us to be informed about that's going on here.
WILL VAN GULIK: I got the version that I worked on that was basically a translation to French, to I think link that also. There is a legal document in German, but I will put that to the mailing list to to the connect list. So at least ‑‑ I think it's a really good thing that's happening there, an interesting one.
REMCO VAN MOOK: All right. Next question from Luca Cicchelli:
"Any ideas about how much paid peering is used in Switzerland?"
WILL VAN GULIK: Well, I can say for myself, because me, as a customer, I do pay for that with one of my companies. So, I think there is ‑‑ I think it's used and I think there is a good service for that, but only from what I know for Swisscom. I don't know that any other big ISP applies anything like that. Basically I think some other one, one said like I'm not going to do any paid peering, I am going to say okay, let's go and you get a transit from me and then it's like not really what people want.
REMCO VAN MOOK: Okay. Next. I am going to close the Q for Q&A right now.
I have a comment and a question from Freddie from Init 7, who knows one or two things about this case.
The first comment is:
"I think it was not highlighted enough that Swisscom and DTAG formed an illegal cartel until January 2016. The most important thing in the court decision is that the Court acknowledges that an ISP has a technical monopoly over its user base. This is new."
I guess I was trying to read the question in that, but those are statements hiding as questions I guess.
So, let me do some admin work here.
WILL VAN GULIK: I can comment on that because I think it was indeed stated in the document, I I agree, because, like, that's ‑‑ well, it's a big detail, let's say, I agree with you Freddie. But, yeah, I think it's ‑‑ it was hard to go and get all the big tricky and interesting part, but I think the cartel part was mentioned and that's indeed something really, really strong, I think in the legal language.
REMCO VAN MOOK: Okay. Then I have Carsten Schiefner:
"Will this have the potential to set precedence for other European countries even when Switzerland is not even an EU country?"
WILL VAN GULIK: I don't know. I mean, it could be something that ‑‑ yeah, I was wondering. It will depend on how it behaves with ‑‑ how the other countries think about that and if they can say well maybe ‑‑ I suspect, let me rephrase that.
I think we could have interesting precedence for other countries to go and force, or like push legal, the incumbent give decent access and like a reasonable price access to their network as a peering.
REMCO VAN MOOK: Then the final one from Arturo:
"Is this the first peering battle going to court or some other that happened in the past that you know of?"
WILL VAN GULIK: So, court, I don't think there is any other. I heard a lot of peering was obviously, but that's ‑‑ I may be mistaken so I don't think I can say on that one. But I don't think there was another court case for that, at least not in Switzerland, not that I'm aware of.
REMCO VAN MOOK: Yeah, I know about a lot of disagreements between large organisations that have had some brushings with courts, but it certainly is the first court verdict I have seen that explicitly refers back to a regulator to set a tariff. So what I would suggest is is that we ‑‑ this is a very recent development, the court judgement was last month, I think, so it's a couple of weeks old. I would suggest that we take this to the next Connect Working Group session to get an update, because I think this is going to be very interesting to follow for all of us.
WILL VAN GULIK: Absolutely. And I would love to have any of the involved parties actually talk about that if they can, because that would be absolutely great, I think, to have their points also on that one.
REMCO VAN MOOK: Okay. So, thank you very much for that, Will. So, round of applause for Will.
(Virtual applause!)
REMCO VAN MOOK: So, next topic, Will, can you introduce that for us, please?
WILL VAN GULIK: Yes. So basically we have got Raphael Maunier and
Benjamin Schilz who will tell about their experience on using remote hands during this Covid‑19 situation. So ‑‑ and they will tell a bit about the story how they deployed it during this period.
I guess the floor is yours, and go ahead.
RAPHAEL MAUNIER: Thank you, Will. You are going to discuss what you asked me about the issue and the challenge we had about deploying a POP for us during Covid‑19. So I will start the talk.
So it was the ‑‑ I want to talk about let's say only technical stuff and we don't ‑‑ we are not doing any marketing stuff because people don't want to experience bits like that.
So basically, why do we decided to go for deploy a POP during Covid‑19. This is a worse, let's say, time in the history to deploy a POP while you can wait at least two or three months. Basically, this was our plan; our plan was deploy a plot in Q2, Q4 this year, and well, you don't expect to have a Covid‑19 coming, so ‑‑
But due to the situation, you will notice that we have a usage increase on traffic, so life doesn't stop even if it's difficult, people need to continue to do their business and our customers as well need to grow. So, they call us and say okay, we need your help. We didn't expect that much, so can you please tell us that you can, let's say, move your plan before Q3.
So, basically we discussed the whole situation and we decided by April 7, that we can do it. So we have to do it and it would be challenging so we make a plan and then to go with the plan.
So what I said. Cannot travel. Cannot access data centres. Cannot order hardware because when you ask Juniper, you ask HP, you ask anyone, okay, can you give me ten servers with one terabit of RAM and everything. They say, okay, so we can deliver, like, late June, maybe July, so basically you have no access to the hardware. What you can do, and most important, you have a risk of your life so you can die from this, so basically ‑‑ anyway, we are challenging people so we accepted and say, okay, let's go for it. And let's make a plan together.
So, what you do when you have a plan. So you need to build a list of what you have to do. So basically, you all know I will not tell you what to do basically, you call the data centre people, you have to negotiate the rates, you have to negotiate everything. Because when you want to do something in Q3, Q4, you know basically of where you are going to go, but it's not done, it's not negotiated.
You have to look at the design, where you are going to connect, and the KAMZ, because most of the time people don't think about it but when you buy two times 100‑gig or something, a lot of the time the logo is okay but when you arrive in the city you have the same fibre contacts and cannot issue. So you have to go with KAMZ, people, can you deliver it or not?
We work as well with the ‑‑ all the IXP to have a peering point because we need connectivity locally.
And also, how we can do it. How we get the hard and all the rest that is not, let's say, easy not to list here, but basically all the stuff like cabling and everything.
So, what we decided to do.
So, as I said, almost impossible to get and to buy a hundred percent of the hardware in time. So what we decided is, okay, we have to reuse some hardware that we have in production, and ‑‑ or in a lab and everything. So hopefully we had some servers in kind of a beta lab, so we have some labs with good hard wears, mostly sometime because it's used to test new stuff. So, we decided okay, at least for two or three months we will use the labs stuff, it's a very good hardware, so we use it, so we get it from the lab, the same as the routing, so we say okay almost impossible to get the gear in three weeks delivered in the data centre so, we decided to use what we call the old version of the hardware we had, so we are using MX 10 K.
Now you have the hardware, you have everything, so you are good.
How you do the staging. What we did was we need to configure it. Everything was delivered to my home basically, in my basement, and my kids and wife were not really happy but had about maybe, I don't know, 24 servers, Juniper, Goodman switches, firewalls, and about 200 cables everywhere running, but it was okay, it was fun, and I also had my kids and my daughter helped me to do the stitching to know where the cable is coming, what is discovered.
What we discovered was the real Internet. It was an interesting event for me as well to show them what we are really doing is not only sitting on a chair and doing some interesting stuff. It was really fun for them to see me doing that. And also, as my son to admit to stack and rack.
Where was that? It was in the great city of Lisbon in Portugal. So, Portugal was just shut down, it was closed. You had no access, you are not allowed to go and I think it's one of the top cities that decided to close the border early. So it was even worse.
In the meantime, you may know that 100 gig is not as big as going to Amsterdam, London and Madrid. We also had this challenge to find some other way of bringing this 100 gig, of course with an affordable price, and with no, let's say, path that can be cut with the fibre cut at the same time.
So, this is how it goes.
So basically, we are using pre‑cabling, so we decided to install everything. As you can see on the right‑hand side, you have ‑‑ we are using fried case to install the hardware and ship them. So I did exactly what I wanted to in the pre‑stitching, so everything was connected, forward, VPN access and everything. So it was fun, and good thing is in France, we have very good Internet connectivity, so I had a sense to have two fibre connections to my home, one of the connections had a public /30, so I was about to put it directly on to the net for VPNs. So basically I was able to work locally and work remotely. So I did connect the directly to the Internet. And this is how we did it.
And everything is done correctly. We spend a lot of time doing a lot of cabling. So, I spent about maybe most of my nights doing it.
And then it was finished. So ready for shipping. So, I packed everything. And then everything was ready to ship. Tested by the team. So I did two or three times I did the ‑‑ to make sure I didn't miss anything or break anything, and then package and ready to go.
BENJAMIN SCHILZ: So now we have everything ready to be shipping to Lisbon, we have to think about how we are going to deploy locally. We pre‑configure all equipment with management IP, with VPN all pre‑configured. And sometimes we have at least one cross‑connect up and running in Lisbon. We should be able to connect to the infrastructure remotely and be able to connect to the equipment to the console ports to connect to the routers, and check that every port that should be up and to debug every connection if needed. So, we just needed one connection, so we had multiple ways to connect to the infrastructure.
So, we can go to the next slide.
We are using about almost the same design all around the world so we have this for easy maintenance and easy deployment. For example, we deploy POPs on the right in the US and 2 POP in Asia and Singapore and in Tokyo just between November and December last year. So we have used to this kind of deployment. But it's our first time to do a full remote deployment, so I think as everybody was already experienced, we managed that in the data centre, we have very good experiences and very bad experiences. So, we are a bit scared about deploying a remote POP in a remote city, we don't have anybody on site, so the to try to work with this work, so this one is our work in the US.
And so we work on step by step process for the locality. So just the timeline.
Everything was cheaper in April 29th, so we really were precious with shipping companies because we had some friends who experienced issues with the fees due to Covid‑19 so we were really cautious about insurance and shipping cost. Everything arrived on the 1st. So it was a first day but Friday was a public holiday, that's why everything was closed most everywhere in Europe, and of course no installation during the weekend. And Monday was a special day in Portugal, so we didn't have enough staff on site to do a deployment like this and work in the data centre at the same time.
So basically we started on Tuesday the 5th, and they used what we build for them, a step by step. It's made by engineers. The good way to describe all the steps and try to not forget anything, so ‑‑ but maximum for two is quite cheap with all the connection. So, for example, on the first page you have the photo of the boxes, they are easy to find in the Equinox warehouse. We can see the boxes and this equipment. How to identify the feed and everything. We tried to maybe 20 pages of instruction, so a step by step thing like building an EKR and stuff.
Just, for example, copper cabling, just connect this port from this equipment to this port. So this is for the fibre, so we are using full cabling of our routers, so basically we have a patch panel with a bunch of fibres and then pure cable, so this is the end of the bunch of cables. I just need to connect the fibre to this router. And after connect all the small fibres and copper cable and at the end of the day, everything should work. So, on Tuesday, they basically rack all the equipment for everything, and we were really lucky because we didn't have any issue during the transport, no damages, no hardware failure.
On Wednesday, they did the copper and fibre cabling of the work. On the Wednesday, they really worked late, a really good local team, they worked late. They tried to connect and cross‑connect, but the port didn't come up, so we tried to fix the issue in two minutes it was fixed and our first 100‑gig backbone was up and running. So we were able to connect to the local POP and check all equipment, check all ports.
And basically, on Thursday it was just a small work like configure the IPU, debug a cross‑connect, check one or two fibres between servers and patch panels. And at the end of day, it was ready. Everything was up. No damages on the hardware. All software up. So we were able to deploy full remote during Covid‑19 with 100‑gig circuits between two countries in less than a month. So it was a really challenging but really interesting experience for the whole team. And by using only remote it was done. We are only using one right now, so it's not 100 percent perfect. Maybe the fibre on the right, I don't know, but everything was working, they did a really good job, cabling and everything.
So interesting experience for the whole team.
RAPHAEL MAUNIER: Basically, I was honestly expressed. We all know the experience with remote ends, all the big guys don't care, all they want to do is send you a bill about cross‑connect and about smart ends.
But honestly, Equinix, I was really surprised by Equinix because they decided to be really up for the solution, most of the fees for installation was not billed to us. Some was, but it was honestly a very good experience to work with this set of people. And the data centre people honestly.
During this period I think Equinix made something that I have never seen them doing really listening to people, trying to help and to find solutions.
So the rack was not really, but they did find a way to support us and to help us in this period.
So, just to recognise the Lisbon team of Equinix are very, very good guys and really focused on customer satisfaction. So, a big thanks to them they deserve to be thanked and I will look forward to working more with them.
So, what we learned from this? Because you need to learn.
So you have to recognise why we had the success.
So the good thing is we, as a company, we decided to start the company as remote first company. So basically we don't care where people are working, we don't care if they are working in Paris, or Leon, or Toulouse, we honestly, the goal is to have people that is correct work remotely and are kind of autonomous in their tasks. So, I think it was a key for us to be able to work, because we designed for that. So, you don't ‑‑ I think you don't expect to have this kind of situation that cannot get out of your home from one day to another day. So, you need to be prepared. So that's my first lesson.
So if we didn't have this before, we would have failed.
Good enough that we had a lot of spare hardware, so, basically we have a spare parts, we are able to build a small POPs to replicate, to mimic what we are doing.
And the good thing is we started to do this kind of installation remotely a long time ago. If we look at the first version of the tasks we started about two years ago on the people alongside from all staff, we expect people to know what they are doing, so we did something that only us can understand. And we tried to improve and we did something, so mostly them did it to have something that is released by step and if it's someone that is not working for us, it's about to do and I think I am glad we did it and we thought about this two years ago. Hopefully we had a good relationship with Equinix and also Kurt, because Kurt was able to deliver everything in a very, very short time and help us to support us. Most people were just looking to help us, so I was really happy to maintain good relationship with them.
And as I said, need for speed, surprise and quality. You need to find a way that you are able to do a good service. So it means that you don't care about ‑‑ you have to care about the customer, of course, but at some point if you need to do quality, your customers are willing to pay for it so it means you have the investment to have hardware ready. It's all a global thing. It's not just about winning the customer over.
And warnings:
So we need to ‑‑ what we did is we need to improve because it's not only beautiful. So, we realise during this time not all the people in the company are able to do this. But it means that mostly three to four people really work on it for about a month with no, almost no sleep. So basically, cannot scale this way. It was okay for this one, it's fun, but let's say we have ten POPs to deploy, it's impossible. Cannot do this if we have ten POPs to deploy. We need to find a way that we need to find a design that we don't need to do a full stitching. Maybe a small shipping of hardware, maybe one rack with out of band connection and everything should be assembled on the site without having us doing staging. We need to do this.
And of course, it's always a sales issue. If the sales did their job before ‑‑ just kidding ‑‑ they should have worked for closely with the customer before to be sure on that do the warnings, because I am pretty sure we would have ‑‑ it would have been possible to have the order done four weeks in order to give it more spare times. So, this is the only thing that we need to improve on our side.
This is it. We tried to do something that is not, let's say, too technical or too marketing and too bullshit, because I don't I don't take bullshit.
That is it. If you have questions.
BENJAMIN SCHILZ: There are two small questions.
REMCO VAN MOOK: First of all, thank you very much, Raphael, and Benjamin for doing this presentation. Well done. Really enjoyed the content of it. I think it's ‑‑ I think this is a struggle that quite a few of us have gone through in recent months. Very happy to see this.
There are a couple of small questions. There is actually three questions, one came in the chat window. There is now four. I am going to disclose the question list now.
"Did you provide all tight screws and little stuff?"
BENJAMIN SCHILZ: Yes, everything, so basically we have a big stock in Paris so we provided everything.
REMCO VAN MOOK: The second question is Andrei:
"If your company is designed as a remote first, where do you keep your lab?
BENJAMIN SCHILZ: We have two labs in Equinix Paris with the lab. So all our equipment. And we have a stock in our office in Notre, Paris suburbs.
REMCO VAN MOOK: Okay. And then I have got Michela Galante ‑‑ a question from Fasilas [phonetic]:
"Thank you, exceptional work. How did you ensure that no back doors accounting with capability or anything fishy hasn't been introduced in the process?"
BENJAMIN SCHILZ: Basically we are doing an equipment with basic configuration, so as soon as everything is connected and we are able to connect everything is ‑‑
RAPHAEL MAUNIER: We are using Ansible so basically if something is shipped and we go through it rewrite them by default with the config that was built before we shipped it. So it's clear we are almost sure that it's not compromised.
REMCO VAN MOOK: Then the final question, again, from Michela, a question from Aleksi Suhonen from TREX ‑‑ I have another one from Andy Davidson I'll get to.
"Are you planning to redo the site with new hardware after restrictions are lifted?"
RAPHAEL MAUNIER: Yes, we plan in September to change it, and to add a second rack, so we have a bunch of ‑‑ it was a basic deployment to do the first POP, to kick off. And then after September we send more servers and we send a new MX 10 K and new servers, etc.
REMCO VAN MOOK: Then, a question from Andy Davidson:
Now that you have prepared such a detailed installation manual, do you intend to always keep it up to date as your equipment evolves and do you think you will do more remote installations out of choice in the future?
BENJAMIN SCHILZ: Yes.
RAPHAEL MAUNIER: Yes. I mean, I will start and you will finish. Basically yes, on the business point of view, bending men ‑‑ business point of the view, yes. The cost of to send people over and to have people not working for a week during the installation, it's ‑‑ it costs more than paying remote. What we are going to do is networking with Equinix and the other guys packets of installation. It will a project, so we have a project of deployment, we have a cost of that, and then as soon as we need to POP we have a cost we know and we ship it. So basically, yes, in future we will not go and deploy the POPs.
REMCO VAN MOOK: Question from Theo Voss:
"What did you do with the flight cases after the installation was completed?"
BENJAMIN SCHILZ: We shipped back to Paris, basically. We just sent the UPS level to the Equinix team and it was just delivered this morning.
REMCO VAN MOOK: That's great. A UPS delivery that actually arrives. That's good. A question from Mattia ‑‑
Waiting for his cross‑connect to Lisbon.
BENJAMIN SCHILZ: It's coming.
REMCO VAN MOOK: Okay. I think that wraps it up for this presentation. Thank you again, Raffaele and Benjamin, for sharing this with us, and I think we are now in time for a coffee break. So, thank you all for joining us, all 427 of you, for this first session, and I hope to see you all in about 13 minutes. Enjoy your coffee break.
(Coffee break)
