Did you actually apply a finish after skipping the grain filler? Calling it "raw" oak isn't right if you used any oil or varnish. That textured look comes from the open pores, but it still needs a sealant to last. I skip filler on my pieces too, but I always use a wipe-on poly to protect the wood.