Disabling interrupts to protect data

On Mon, 26 Oct 2009 13:30:42 -0500, Vladimir Vassilevsky
<nospam@nowhere.com> wrote:

>> A common approach to providing an atomic operation.
>> Some CPUs don't need this.
>
>Some OSes claim that they never disable interrupts, however from what I 
>have seen it was all very impractical.

And some explain it with the longest un-interruptible instruction such
as saving or storing a bunch of registers. 
-- 
42Bastian
Do not email to bastian42@yahoo.com, it's a spam-only account :-)
Use <same-name>@monlynx.de instead !

42Bastian Schick wrote:
> On Mon, 26 Oct 2009 12:19:02 -0700, D Yuniskis
> <not.going.to.be@seen.com> wrote:
> 
> 
>>FreeRTOS info wrote:
>>
>>>D Yuniskis wrote:
>>>
>>>>and then "schedule" a defered activation.  So, the jiffy
>>>>terminates as expected.  The interrupted routine (probably
>>>>an OS action) finishes up what it was working on, then,
>>>>examines a flag to see if it can "simply return" or if it has
>>>>to process some deferred "activity" 
>>>
>>>....and how are you protecting access to the flag - or are you assuming 
>>>the hardware supports atomic read-modify-writes on variables - or that 
>>>the hardware supports atomic semaphore type operations?
>>
>>Assuming you don't have a second processor...
>>
>>ever hear of a "Test and Set" instruction?
> 
> 
> "Test and set" or how you name it is impractical in a interrupt
> context. You just can't loop and wait for the semaphore.

Yes. Also test-set is not suitable when manipulating stack and frame 
pointers or dealing with nested interrupts.


Vladimir Vassilevsky
DSP and Mixed Signal Design Consultant
http://www.abvolt.com

42Bastian Schick wrote:
> On Mon, 26 Oct 2009 21:35:11 +0100, David Brown
> <david.brown@hesbynett.removethisbit.no> wrote:
> 
>> I suspect that Richard has looked very carefully through a very large 
>> number of instruction sets looking for exactly that sort of instruction...
>>
>> There are certainly plenty of architectures that don't have test-and-set 
>> instructions, or anything similar.  As a general rule, test-and-set type 
>> instructions don't fit well with the ethos of RISC, so RISC processors 
> 
> Hmm, I wonder which.
> 
>> typically don't have such instructions.  They either rely on disabling 
>> interrupts (common for microcontrollers and smaller processors, where 
>> this is a simple task), or have more sophisticated mechanisms such as 
>> "reservations" used by the PPC processors.
> 
> Yeah, but don't rely on it. I had to re-write great parts of my RTOS
> because Freescale limited the use of the reservation e.g. in MPC55xx
> that it is not usable anymore.

Reservations are hardware intensive to implement, unless you are willing 
to pay extra memory latency.  But they scale well for multi-processor 
systems, making them a good solution for "standard" PPC processors.  The 
PPC core in the MPC55xx is more like a microcontroller core than a "big" 
processor core, and Freescale probably saved a fair amount of silicon 
and complexity by dropping reservations so that you must use traditional 
interrupt disabling.  I think it is the right choice for these devices, 
but then I haven't been trying to write optimal RTOS code that is 
portable across different PPC devices!

D Yuniskis wrote:
> Vladimir Vassilevsky wrote:
>> Some OSes claim that they never disable interrupts, however from what
>> I have seen it was all very impractical. Once you have more or less
>> sophisticated structure of threads and interrupts, you've got to have
>> critical parts with the interrupts disabled.
> 
> You only need to disable interrupts if an interrupt context
> can access those "shared objects" *without* observing whatever
> other "mutex" mechanism you are using.
> 
> It *can* be done.  But, it is a lot trickier than just
> a tiny little critical region.

Yep.

> E.g., if the jiffy comes along (perhaps the most notable
> active element that *would* be interrupt spawned and asynchronously
> compete for access to those strctures), it has to notice that a
> critical region has been entered (by whatever it has interrupted!)
> and then "schedule" a defered activation.  So, the jiffy
> terminates as expected.  The interrupted routine (probably
> an OS action) finishes up what it was working on, then,
> examines a flag to see if it can "simply return" or if it has
> to process some deferred "activity" (i.e. those things that the
> jiffy *would* have done had it been fortunate enough to come
> along "outside" that critical region.

The problem here is how you "schedule" the deferred activity. You can
make an array of sig_atomic_t variables, one for each possible activity,
the interrupt sets them, the kernel exit checks and clears them. But
this obviously does not scale.

When you make something generic, you'll start needing linked lists of
stuff where you put the deferred actions in. So you now need some kind
of atomic link-creation or swap routine. Okay, this can also be
implemented without disabling interrupts, by having the interrupt
routine detect whether it interrupted the atomic routine. It's possible,
but it's easy to get wrong.

It's much simpler to just wrap the three statements into a CLI/STI.
In particular when your processor has multi-cycle instructions which
implicitly block interrupts for a dozen cycles while they execute - so
why should these be permitted and a three-cycle CLI/STI should not?


  Stefan

Stefan Reuther wrote:
> D Yuniskis wrote:
> 
>> E.g., if the jiffy comes along (perhaps the most notable
>> active element that *would* be interrupt spawned and asynchronously
>> compete for access to those strctures), it has to notice that a
>> critical region has been entered (by whatever it has interrupted!)
>> and then "schedule" a defered activation.  So, the jiffy
>> terminates as expected.  The interrupted routine (probably
>> an OS action) finishes up what it was working on, then,
>> examines a flag to see if it can "simply return" or if it has
>> to process some deferred "activity" (i.e. those things that the
>> jiffy *would* have done had it been fortunate enough to come
>> along "outside" that critical region.
> 
> The problem here is how you "schedule" the deferred activity. You can
> make an array of sig_atomic_t variables, one for each possible activity,
> the interrupt sets them, the kernel exit checks and clears them. But
> this obviously does not scale.

Append a pointer to the ASR that needs to be deffered to
the end of a list of "deffered procedure calls".  When the
ISR returns, the OS schedules this "list" of DPC's (in
sequential order).

Of course, you still have to guarantee that you have enough
"overall" time to handle the tasks at hand, lest this list
grow indefinitely.  (i.e., your design has to "work"!  :> )

> When you make something generic, you'll start needing linked lists of
> stuff where you put the deferred actions in. So you now need some kind
> of atomic link-creation or swap routine. Okay, this can also be
> implemented without disabling interrupts, by having the interrupt
> routine detect whether it interrupted the atomic routine. It's possible,
> but it's easy to get wrong.

This last point is why folks seem to resort to the more heavy-handed
approach of unilaterally disabling interrupts for *everything*
that is "hard" or "easy to get wrong".

> It's much simpler to just wrap the three statements into a CLI/STI.
> In particular when your processor has multi-cycle instructions which
> implicitly block interrupts for a dozen cycles while they execute - so
> why should these be permitted and a three-cycle CLI/STI should not?

There is nothing wrong with disabling interrupts!  But, like
everything else, you have to understand *why* you are doing
it and when it is *appropriate* to do so.

In my experience, it seems that people turn interrupts off
"too early" in a code fragment and turn them back on "too late".
Often, if you look at the code, you can see things that could
have been moved outside of the critical region (sometimes with
some added complexity) to reduce the size of that region.
This usually results in "needing more CPU" than you *really* do.

*If* your system can handle high interrupt latencies, then
you can "afford" to disable them for longer periods of
time.  But, you have to make sure you know just how long
those periods can *become* -- since a deferred IRQ is
more likely to encounter some *other* IRQ during its execution
(i.e., you may return from one ISR just to find yourself
in *another*, etc.)

You also have to be careful about how you apply the DI/EI
idiom.  If, for example, the code you are executing can
take place *within* a region where interrupts are off, then
you want to be sure you don't blindly re-enable them at
the end of "this" critical region only to (belatedly)
discover that you have corrupted some other *encasing*
critical region.  I.e., you may have to examine the state
of the interrupt mask/level before disabling so that you can
correctl restore it afterwards.

The same is true of *all* locking mechanisms.  E.g., taking a lock
on a file before you *really* need it, etc. (especially if that
"file" is a *special* file!)

D Yuniskis wrote:
> Stefan Reuther wrote:
>> D Yuniskis wrote:
>>> E.g., if the jiffy comes along (perhaps the most notable
>>> active element that *would* be interrupt spawned and asynchronously
>>> compete for access to those strctures), it has to notice that a
>>> critical region has been entered (by whatever it has interrupted!)
>>> and then "schedule" a defered activation. [...]
>>
>> The problem here is how you "schedule" the deferred activity. You can
>> make an array of sig_atomic_t variables, one for each possible activity,
>> the interrupt sets them, the kernel exit checks and clears them. But
>> this obviously does not scale.
> 
> Append a pointer to the ASR that needs to be deffered to
> the end of a list of "deffered procedure calls".  When the
> ISR returns, the OS schedules this "list" of DPC's (in
> sequential order).

Exactly this "append a pointer" thing is where an atomic operation can
come in very handy. You shouldn't append to the list while it's being
processed, nor should you interrupt another instance in another
interrupt (if you have nested interrupts).

All this is totally simple when you can use CLI/STI:
   void atomic_swap(node** a, node** b, node* c) {
      cli(); *a = *b; *b = c; sti();
   }

   // prepend new node to a list (easier than appending...)
   node* list;
   node* newNode;
   atomic_swap(&newNode->next, &list, newNode);

   // get list (for processing) and make it empty
   node* oldList;
   atomic_swap(&oldList, &list, 0);
(some strategically-placed 'volatile' can be appropriate.)

>> When you make something generic, you'll start needing linked lists of
>> stuff where you put the deferred actions in. So you now need some kind
>> of atomic link-creation or swap routine. Okay, this can also be
>> implemented without disabling interrupts, by having the interrupt
>> routine detect whether it interrupted the atomic routine. It's possible,
>> but it's easy to get wrong.
> 
> This last point is why folks seem to resort to the more heavy-handed
> approach of unilaterally disabling interrupts for *everything*
> that is "hard" or "easy to get wrong".

Actually I see much more often that people do not use proper locking and
ponder about the occasional crashes that happen once in a fortnight :-)

By "easy to get wrong" I mean, for example, that you have to tell all
your interrupt routines about your atomic_swap routine, so that they can
see that they've interrupted it and can complete or restart it. This is
nasty asm trickery. It enlarges all your interrupts by a dozen
instructions (at least some compares and jumps). And all that just
for a sticker "never disables interrupts" in the glossy brochures?
Sorry, I'd rather take the additional three-cycle interrupt latency for
running atomic_link under CLI.

>> It's much simpler to just wrap the three statements into a CLI/STI.
>> In particular when your processor has multi-cycle instructions which
>> implicitly block interrupts for a dozen cycles while they execute - so
>> why should these be permitted and a three-cycle CLI/STI should not?
> 
> There is nothing wrong with disabling interrupts!  But, like
> everything else, you have to understand *why* you are doing
> it and when it is *appropriate* to do so.

I'm quite optimistic that I understand :-)


  Stefan

Stefan Reuther wrote:
> D Yuniskis wrote:
>> This last point is why folks seem to resort to the more heavy-handed
>> approach of unilaterally disabling interrupts for *everything*
>> that is "hard" or "easy to get wrong".
> 
> Actually I see much more often that people do not use proper locking and
> ponder about the occasional crashes that happen once in a fortnight :-)
> 
> By "easy to get wrong" I mean, for example, that you have to tell all
> your interrupt routines about your atomic_swap routine, so that they can
> see that they've interrupted it and can complete or restart it. This is
> nasty asm trickery. It enlarges all your interrupts by a dozen
> instructions (at least some compares and jumps). And all that just
> for a sticker "never disables interrupts" in the glossy brochures?
> Sorry, I'd rather take the additional three-cycle interrupt latency for
> running atomic_link under CLI.

You can wrap all of your interrupts with a common dispatch routine.
But, it boils down to the same thing:  there is more to get done.

The one thing in your favor is that these ISR's are, in theory,
relegated to the operating system's responsibility.  It is
a fair bit harder when you let "user code" migrate into this
realm.

[which is another aspect of this:  folks who do things *in* ISRs
that don't need to be handled there -- resulting in bloated ISRs
which impacts the responsiveness of the whole system... which
causes folks to tend to put more in *other* ISRs... which...]

>>> It's much simpler to just wrap the three statements into a CLI/STI.
>>> In particular when your processor has multi-cycle instructions which
>>> implicitly block interrupts for a dozen cycles while they execute - so
>>> why should these be permitted and a three-cycle CLI/STI should not?
>> There is nothing wrong with disabling interrupts!  But, like
>> everything else, you have to understand *why* you are doing
>> it and when it is *appropriate* to do so.
> 
> I'm quite optimistic that I understand :-)

Stefan Reuther wrote:

> By "easy to get wrong" I mean, for example, that you have to tell all
> your interrupt routines about your atomic_swap routine, so that they can
> see that they've interrupted it and can complete or restart it. This is
> nasty asm trickery. It enlarges all your interrupts by a dozen
> instructions (at least some compares and jumps). And all that just
> for a sticker "never disables interrupts" in the glossy brochures?
> Sorry, I'd rather take the additional three-cycle interrupt latency for
> running atomic_link under CLI.

A related problem is where a single shared variable at driver level that 
needs to be accessed from different functions, all of which disable the 
interrupt within the critical section. Disabling interrupts is fine, but 
it is necessary to know at the time of reenabling any instance, what the 
current interrupt state was for the device, prior to the call. The 
solution was to write a couple of functions to stack the current device 
interrupt state before disabling, then pop on exit, rather than just a 
hard disable / reenable.

I suppose this will turn out to be a standard technique, but new to me 
at least...

Regards,

Chris

ChrisQ wrote:

> A related problem is where a single shared variable at driver level that 
> needs to be accessed from different functions, all of which disable the 
> interrupt within the critical section. Disabling interrupts is fine, but 
> it is necessary to know at the time of reenabling any instance, what the 
> current interrupt state was for the device, prior to the call. The 
> solution was to write a couple of functions to stack the current device 
> interrupt state before disabling, then pop on exit, rather than just a 
> hard disable / reenable.
> 
> I suppose this will turn out to be a standard technique, but new to me 
> at least...
> 
> Regards,
> 
> Chris

...and also a technique that won't work on most compiler/architecture 
combinations as modifying the stack in this way will prevent the code in 
the critical region from executing correctly (any stack pointer relative 
addressing just 'aint going to work).

-- 

Regards,
Richard.

+ http://www.FreeRTOS.org
Designed for Microcontrollers.  More than 7000 downloads per month.

+ http://www.SafeRTOS.com
Certified by T�V as meeting the requirements for safety related systems.

FreeRTOS info wrote:
> ChrisQ wrote:
> 
>> A related problem is where a single shared variable at driver level 
>> that needs to be accessed from different functions, all of which 
>> disable the interrupt within the critical section. Disabling 
>> interrupts is fine, but it is necessary to know at the time of 
>> reenabling any instance, what the current interrupt state was for the 
>> device, prior to the call. The solution was to write a couple of 
>> functions to stack the current device interrupt state before 
>> disabling, then pop on exit, rather than just a hard disable / reenable.
>>
>> I suppose this will turn out to be a standard technique, but new to me 
>> at least...
>>
>> Regards,
>>
>> Chris
> 
> ...and also a technique that won't work on most compiler/architecture 
> combinations as modifying the stack in this way will prevent the code in 
> the critical region from executing correctly (any stack pointer relative 
> addressing just 'aint going to work).
> 

No, not using the system stack. By writing functions, I mean functions 
using an short byte array to simulate a stack...

Regards,

Chris