Does ethernet switch quality matter when using fiber connection?

wall > standard ethernet cable to switch > lumin x1 via fiber. basic setup

When you say "wall" you mean coming out of your router, correct?

Most routers output through Ethernet and most streamers/servers input through Ethernet.  Therefore, in most cases, people need two conversions.  If your Lumin has a direct optical input then you have an advantage by needing one less convertor.

Some folks convert the Ethernet to fiber for only a short distance just ahead of the streamer/server as sort of a galvanic break.  There are companies who make small boxes that filter and do other magical things and many of those also provide  galvanic isolation to reduce/eliminate EMI/RFI being transmitted to the streamer/server.  I convert to fiber imediately after the router and then again just before the streamer/server.  Most of my 45 foot run is in fiber.  Honestly, I cannot hear a difference whether that run is Ethernet cable or fiber but I have left the fiber in-place.

Regarding your switch, @audiom3 nailed it with the importance of using a LPS on the switch (and on the Ethernet to Fiber converters).  Regarding the purchase of an "audiophile approved" switch, only you can decide how important that is to the sound of your system.  There are people here who are pretty passionate about their switches with the English Electric 8 switch and the Bonn N8 Silent Angel switches both being popular.   OTOH, you might look at this.

Some other stuff you might want to read about includes the Network Acoustics ENO filter and the Intona Technology USB Hi-Speed Isolator.  The level of impact of certain items on the sound of your system may be affected by the design of your streamer/server, the EMI/RFI in your ambient environment, and other factors.  The best way to know for sure is to listen.

Is it still worth investing in an audiophile grade switch and if so why?

No ,There is never a reason to buy a so called "audiophile grade switch" . Learn what a switch does  instead, then you have at least furthered your knowledge. 

My recent experience with an audiophile switch disappointing. I have no filtering of any kind prior to router, Audioquest Vodka feeding router from modem and router to server. LPS on router. All my network optimization is post server. Over time I became convinced audiophile switch prior to server would improve sq. Quite the contrary, i much prefer no switch, seems my router with lps is providing everything I need.

My suspicion is the switch and its internal clock was one clock too many in my setup. My switch also had built in very nice quality lps. YMMV. I'll also second use of fiber conversion, that was a very nice additon!

I recently bought a CA streamer that after week stopped recognizing my NAS.  CA told me to change the switch and that solved it.  Not a sonic issue, but obviously an important one