Recently, I started a project in which the main goal is to wrap vendor specific drivers into a HAL so that I can use them among of vendor chips, I have it nailed pretty much down; however, I am now stepping into the game of API designing and I do not really understand how the procedure should follow. For instance, 

I have an RGBLED API that contains an LED struct for accessing LED functions. I have an init function that attaches the BSP layers PWM modules to an RGB LED struct so that I can adjust the duty cycles. However, I'm a little confused about controlling the actual hardware. Ideally I want to run a simple OS that can call an application and the application is completely abstracted from any hardware it just uses the APIS to make things happen. Would the OS be responsible for calling the HAL drivers to update the duty cycle or would that instead be inside the actual RGB LED API? 

Another example is the serial API. Would the serial API be responsible of using the HAL UART drivers library to store values into a circular buffer or should the OS be doing that? 

I guess my overarching question is, how do BSP, HAL, and OS interact with each other? 

how these interact depends on the OS and it's philosophy of things and what you think of as "embedded" (a subject that I have been having "discussions" about for some time now).

Example: FreeRTOS (and other actual embedded OSs) could care less about your hardware.  In that OS, your layers call the HAL drivers.  Zephyr, cares deeply about your hardware and wants to call that stuff on its own (you apparently can call HAL things yourself, but the OS environment discourages it).

Why? It depends on where you're coming from. FreeRTOS (and a host of others) come from small iron and move upwards in scale whereas Zerphyr (and a few like it) comes from a Big Iron approach and attempts to move downwards.

IMHO (and a position I've argued a lot about of late) is that if you don't care deeply about the hardware you're on, you're not doing embedded programming; you're doing captive programming (a term I'm coining - basically headless desktop programming).

So, how you approach this depends deeply on your outlook of iron. For my part, I'm an old grey iron monger and my approach is keeping as close as reasonable to the iron, becoming one with it.  Many, many places these days are not in that camp and think that hardware is irrelevant. 

YRMV

It's such a grey area now on what to think of "embedded" as. I have just been struggling with the concept of how some companies/individuals can easily port different vendor MCU's to their projects without needs of completely rewriting code. My biggest questions being: 

1. Should I create a generic driver interface that wrap drivers from vendor specific code?
2. How do I organize a BSP layer? Should I use only driver interfaces or should I be okay to use vendor specific drivers? 

My general fear is to develop a system that can only be used with one MCU. 

It is also a struggle because it seems the whole community does not agree where HAL layer begins/ends.

Why do you fear to develop a firmware for one MCU only? Are you writing a library?

From my experience, it is not very common to change the MCU vendor for a product, unless there are some real chip shortage problems, or the requirements change drastically. Anyways, trying to support all possible vendors will probably be more time consuming and costly than porting the code later on. 

Nevertheless, careful design of the software architecture is always important, and using interfaces is a good place to start to reduce coupling between modules, not only for porting.

@Emiled is right, unless you are writing a cross platform MCU library, you are better off writing your API the BSP / HAL you have, and cross that bridge when you get to it. 

If you have wrapped your hardware interfaces in higher level functions, and you keep all your HAL references in there, it makes porting or reuse easy and contained if / when you switch vendors. That is useful but not overly complicated abstraction.  

As you can see in many open source projects, making it truly hardware independent is quite painful, since it makes it hard to follow, uses lots of preprocessor, typically needs cooperating teams, and multiples your test requirements.

BSPs are specific collections of configured drivers for specific boards which include a particular MCU already. So it also contains vendor specific drivers, and those for off-board peripherals. It should be between the HAL and your API, providing a more coherent view of all the board's capabilities. 

I've seen a lot of companies waste a lot of time on projects like this. The successful ones write the adaptation layer in a HLL like C or C++ and map that to whatever drivers, framewoks, HAL layers that the chip vendor/RTOS supplies. It's fast and easy and applications become relatively easy to port to new hardware and software environments.

They adapt that HLL interface to whatever they need on a a projct - to the specific RTOS layer and whatever HAL and drivers are needed for that job.  

Trying to adapt the HAL from Renesas, STM, NXP, etc to a universal HAL layer or or a universal BSP is a waste in my opinion because it adds code, extra layers of abstraction, and makes for bloated, difficult to understand and test code.

But if you're going to try something, I'd standarize on the Linux style API for IO (open, close, read, write, and IOCTL) with maybe some Posix-like calls for special functions. 

Could you expand on the adaptation layer? I understand that the HAL is primarily composed of vendor drivers, etc. But isn't what I'm doing just expanding the HAL? I'm just wrapping vendor specific drivers in C already. Or is what I'm doing more correlate to the adapation layer? 

You are free to call OS functions in your HAL and BSP implementations. For instance for your serial driver, the rx interrupt can write received bytes to a message queue. Suppose you have a uint32_t uart_get_bytes(uint8_t* p_buffer, const uint32_t buffer_length) function in your serial API: in the implementation, you can copy up to buffer_length bytes from the queue to the provided buffer. 

The same goes for your BSP implementation. You can for instance call OS functions to wait for a semaphore indicating that a serial transfer was complete. Conceptually, the OS is more of a service to be used by everyone. The BSP layer is above the HAL, so it can use HAL functions. On the other hand, the HAL implementation should use only OS functions and functions provided by the vendor.

In summary:

• OS: can be used by any module, it may need HAL functions
• BSP: uses HAL and OS
• HAL: uses OS and vendor files
• Application : this is your business logic, and should not call HAL functions anymore, but rather use the BSP layer (and OS of course).

On a more general note, try to keep it simple and do not over-engineer your code. Using a HAL and BSP is good, but perhaps you do not really need to separate the two. Simply de-couple your business logic (application) from the underlying infrastructure by defining an API that covers only what you need for you specific project and leave the other stuff aside. If in a further project you need more capabilities, you can always extend your API.

For this to work, it is helpful to adopt a "top-bottom" approach by designing the APIs based on the application needs (versus a more traditional "bottom-top" approach, where your bottom layers define how your application has to use them). For instance, if you only need to send out debug information on the UART, why bother implementing the receive part of the driver? Or if you have an RTC chip which support calendar, alarms, etc. but you only need to read the time, implement functions that read only the time from the RTC. If a later project needs the calendar, implement a new function to read the date. It is simpler, faster, and cheaper to implement a driver based on what is really needed instead of "oh but this could be useful", "do we have to support this peripheral's feature?", etc.